graphviper.graph_tools.generate_airflow_workflow

Functions

generate_airflow_workflow(viper_graph[, dag_id, ...])

Generate an Airflow DAG Python source file from a viper map/reduce graph.

airflow_dag_to_graphviz(dag)

Convert an Airflow DAG to a graphviz Digraph object.

Module Contents

generate_airflow_workflow(viper_graph, dag_id='0', schedule_interval=None, filename='airflow_dag_test.py', dag_name='map_reduce')[source]

Generate an Airflow DAG Python source file from a viper map/reduce graph.

Deprecated since version The: Airflow backend is deprecated and will be removed in a future release. Use graphviper.graph_tools.generate_dask_workflow.generate_dask_workflow() or graphviper.graph_tools.process_with_mpi.processes_with_mpi() instead.

The map node task (and, if present, the reduce node task) are extracted with inspect.getsource() and written into a standalone Airflow DAG module at filename. Only mode="single_node" reduce graphs are supported.

Parameters:

  • viper_graph (Dict) – Graph produced by graphviper.graph_tools.map.map() (and optionally graphviper.graph_tools.reduce.reduce()).

  • dag_id (str, optional) – Currently unused; retained for API compatibility, by default "0".

  • schedule_interval (optional) – Currently unused; retained for API compatibility, by default None.

  • filename (str, optional) – Path of the Airflow DAG Python file to write, by default "airflow_dag_test.py".

  • dag_name (str, optional) – Name of the generated @dag function, by default "map_reduce".

Returns:

The function writes the generated DAG source to filename as a side effect and returns nothing.

Return type:

None

Raises:

AssertionError – If a reduce stage is present with a mode other than "single_node".

airflow_dag_to_graphviz(dag)[source]

Convert an Airflow DAG to a graphviz Digraph object.

Deprecated since version The: Airflow backend is deprecated and will be removed in a future release.

Parameters:

dag (airflow.models.DAG) – The Airflow DAG object.

Returns:

A graphviz.Digraph object representing the DAG.

Return type:

graphviz.Digraph