You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I use athena.add_partition>: to make partition in hourly. So calc current date & hour from session_time by digdag's feature.I guess other ops such as ctas , query ,,, supports nested variables, but add_partition (and maybe drop_partiton) doesn't work well.
In the my example.dig, thrown exception about s3 path, but in our production environment, the partition was created with the value that ${moment(session_time...} was not expanded.
example.dig
If you set ${ny_dt} insted of ${curr_dt} at partition_kv: work normaly.
_export:
plugin:
repositories:
- file://${repos}# - https://jitpack.iodependencies:
- pro.civitaspo:digdag-operator-athena:0.4.0athena:
auth_method: profilecurr_dt: ${moment(session_time).format("YYYY-MM-DD")}ny_dt: 2020-01-01+step7:
athena.ctas>: select 1 as a, 2 as b, '${ny_dt}' as dt union all select 4 as a, 5 as b, '${curr_dt}' as dtdatabase: ${database}table: hogelocation: ${output}partitioned_by: [dt]# +step8:# athena.drop_partition>:# database: ${database}# table: hoge# partition_kv:# dt: "${curr_dt}"+step9:
athena.add_partition>:
database: ${database}table: hogepartition_kv:
dt: "${curr_dt}"
error
* +example+step9:
Illegal character in path at index 37: s3://imaoka-oregon-athena/digdag/dt=${moment(session_time).format("YYYY-MM-DD")}/ (illegal argument)
The text was updated successfully, but these errors were encountered:
Hi, I use
athena.add_partition>:
to make partition in hourly. So calc current date & hour fromsession_time
by digdag's feature.I guess other ops such asctas , query ,,,
supports nested variables, but add_partition (and maybe drop_partiton) doesn't work well.In the my example.dig, thrown exception about s3 path, but in our production environment, the partition was created with the value that ${moment(session_time...} was not expanded.
example.dig
If you set ${ny_dt} insted of ${curr_dt} at
partition_kv:
work normaly.error
The text was updated successfully, but these errors were encountered: