Re:Lessons learned from running Apache Airflow at scale
by artwr on Monday May 23, 2022 @03:25PM
(#31483782)
`your_task.override(task_id="your_generated_name")` not working for you?
[ Parent
| on HN ]
Re:Lessons learned from running Apache Airflow at scale
by mywittyname on Monday May 23, 2022 @04:08PM
(#31484204)
I got pretty excited when I read this response, but no, it doesn't work. I'm not sure how this would work since annotated tasks return an xcom object.
Can you point me to the documentation on this function? It's possible I'm not using it correctly.
I can do something like this, which works locally, but breaks when deployed:
res = annotated_task_function(...)
res.operator.task_id = 'manually assigned task id'
Re:Lessons learned from running Apache Airflow at scale
by flowair on Monday May 23, 2022 @04:23PM
(#31484372)
@task.python(task_id="this_is_my_task_name")
def my_func():
...
Re:Lessons learned from running Apache Airflow at scale
by mywittyname on Monday May 23, 2022 @05:33PM
(#31485033)
This still has the problem that, when you call my_func multiple times in the same dag, the resulting tasks will be labelled, my_func, my_func__1, my_func__2, ...