graphviper.graph_tools.reduce

Attributes

REDUCE_MODES

Functions

reduce(→ Dict)

Appends a reduce step to the graph created by the graphviper.graph_tools.map(). function.

Module Contents

REDUCE_MODES = ('tree', 'single_node', 'tree_n')[source]
reduce(graph: Dict, reduce_node_task: Callable[Ellipsis, Any], input_params: Dict, mode: {'tree', 'single_node', 'tree_n'} = 'tree', n_batch: int = 2) → Dict[source]

Appends a reduce step to the graph created by the graphviper.graph_tools.map(). function.

Parameters:

  • graph (Dict) – Graph produced by graphviper.graph_tools.map().

  • reduce_node_task (Callable[..., Any]) – The function that forms the nodes in the reduce portion of the graph must have two parameters: input_data and input_params. The input_data represents the output from the mapping nodes, while input_params comes from the reduce parameter with the same name.

  • input_params (Dict) – The input parameters to be passed to node_task.

  • mode ({"tree","single_node","tree_n"}, optional) –

    • single_node: The output from all map nodes is sent to a single node,

    • tree: The outputs are combined using a binary tree reduction (each reduce node combines two inputs), by default “tree”.

    • tree_n: The outputs are combined using a tree reduction in which each reduce node combines n_batch inputs per layer, repeating layer by layer until a single node remains. n_batch=2 reproduces tree; a large n_batch reduces the number of layers (shallower tree, more inputs combined per node) trending toward single_node behaviour.

  • n_batch (int, optional) – Number of inputs combined by each reduce node, per layer, when mode="tree_n". Must be >= 2 (validated unconditionally so a meaningless value can never be recorded on the graph). Used only by mode="tree_n"; the other modes ignore it. Default 2.

Returns:

The input graph dict with an added "reduce" entry ({"mode", "node_task", "input_params", "n_batch"}) describing the reduce step. No dask.delayed objects are constructed here; the actual reduce tree is built later by graphviper.graph_tools.generate_dask_workflow() (or executed by graphviper.graph_tools.processes_with_mpi()).

Return type:

Dict

Raises:

ValueError – If mode is not one of {"tree", "single_node", "tree_n"}, or if n_batch is not an integer >= 2 (bool is rejected too).

Notes

The chosen mode/n_batch are recorded on the graph and honoured by both execution backends: graphviper.graph_tools.generate_dask_workflow() (builds the corresponding dask.delayed reduce tree) and graphviper.graph_tools.processes_with_mpi() (applies the same fan-in when reducing the gathered map results).