Is there a benefit to use the "with dag as DAG(...)" clause to create a DAG

Question

I create my Airflow DAGs as follows:

dag = DAG(...)

But in multiple tutorials and course I see that they use the with ... as clause like this:

with DAG(...) as dag:
    # Code that will use the dag variable.

I guess this way, the DAG instances will be destroyed after the scheduler executes the code block, but is there a real benefit in doing so? I can't find any documentation talking about this.

You can see what `__enter__` and `__exit__` do in the source code: https://github.com/apache/airflow/blob/master/airflow/models/dag.py — jonrsharpe, Feb 07 '20 at 13:32
@jonrsharpe I see that the the dag is pushed to a dag context manager, which seems to be some kind of collection. Does that mean are supposed to use the "with ... as " clause when creating a dag ? — Robert Reynolds, Feb 07 '20 at 13:35
Reason is mentioned [here](https://stackoverflow.com/a/1369553/10981911) — manesioz, Feb 07 '20 at 14:04
I understand what the "with as" clause do but I was rather asking the impact on airflow ecosystem. — Robert Reynolds, Feb 07 '20 at 14:06
One little advantage is that you don't have to type in each task dag=dag — Javier Lopez Tomas, Feb 07 '20 at 14:31

Forest · Accepted Answer · 2022-01-26T04:12:22.717

6

Yes.

If you understand what with...as does, then you should understand that its impact on the airflow ecosystem is really no different.

Specifically, it ensures that unmanaged resources -in this case implementations of the DAG class- are properly cleaned up, even if there are exceptions thrown (without needing to use a try/except block every time.)

Additionally it's nice to not have to add dag=dag to every single one.

edited Jan 26 '22 at 04:12

answered May 18 '20 at 18:11

Forest

808
6
9

What actually needs to be cleaned up, though? `with` is used a lot to make sure that open file handles are automatically closed, for example. What would an Airflow DAG need to make sure was cleaned up? – Stephen Oct 13 '21 at 13:48

Is there a benefit to use the "with dag as DAG(...)" clause to create a DAG

1 Answers1