Skip to content

Nesting

As seen from the definitions of parallel or map, the branches are pipelines themselves. This allows for deeply nested workflows in runnable.

Technically there is no limit in the depth of nesting but there are some practical considerations.

  • Not all workflow engines that runnable can transpile the workflow to support deeply nested workflows. AWS Step functions and Argo workflows support them.

  • Deeply nested workflows are complex to understand and debug during errors.

Example

You can run this pipeline by python examples/06-parallel/nesting.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
"""
Example to show case nesting of parallel steps.

runnable does not put a limit on the nesting of parallel steps.
Deeply nested pipelines can be hard to read and not all
executors support it.

Run this pipeline as:
    python examples/06-parallel/nesting.py
"""

from examples.common.functions import hello
from runnable import NotebookTask, Parallel, Pipeline, PythonTask, ShellTask, Stub


def traversal(execute: bool = True):
    """
    Use the pattern of using "execute" to control the execution of the pipeline.

    The same pipeline can be run independently from the command line.

    WARNING: If the execution is not controlled by "execute", the pipeline will be executed
    even during the definition of the branch in parallel steps.
    """
    stub_task = Stub(name="hello stub")

    python_task = PythonTask(
        name="hello python",
        function=hello,
    )

    shell_task = ShellTask(
        name="hello shell",
        command="echo 'Hello World!'",
    )

    notebook_task = NotebookTask(
        name="hello notebook",
        notebook="examples/common/simple_notebook.ipynb",
        terminate_with_success=True,
    )

    # The pipeline has a mix of tasks.
    # The order of execution follows the order of the tasks in the list.
    pipeline = Pipeline(steps=[stub_task, python_task, shell_task, notebook_task])

    if execute:  # Do not execute the pipeline if we are using it as a branch
        pipeline.execute()

    return pipeline


def parallel_pipeline(execute: bool = True):
    parallel_step = Parallel(
        name="parallel step",
        terminate_with_success=True,
        branches={
            "branch1": traversal(execute=False),
            "branch2": traversal(execute=False),
        },
    )

    pipeline = Pipeline(steps=[parallel_step])

    if execute:
        pipeline.execute()
    return pipeline


def main():
    # Create a parallel step with parallel steps as branches.
    parallel_step = Parallel(
        name="nested_parallel",
        terminate_with_success=True,
        branches={
            "branch1": parallel_pipeline(execute=False),
            "branch2": parallel_pipeline(execute=False),
        },
    )

    pipeline = Pipeline(steps=[parallel_step])
    pipeline.execute()
    return pipeline


if __name__ == "__main__":
    main()

You can run this pipeline by runnable execute examples/parallel/nesting.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
branch: &simple_branch
  description: |
    Use this pattern to define repeatable branch

    This pipeline is similar to one defined in:
      examples/02-sequential/traversal.yaml
  start_at: hello stub
  steps:
    hello stub:
      type: stub
      next: hello python
    hello python:
      type: task
      command_type: python
      command: examples.common.functions.hello # dotted path to the function.
      next: hello shell
    hello shell:
      type: task
      command_type: shell
      command: echo "Hello World!" # Command to run
      next: hello notebook
    hello notebook:
      type: task
      command_type: notebook
      command: examples/common/simple_notebook.ipynb # The path is relative to the root of the project.
      next: success


# This branch is similar to a branch parallel.yaml
nested_branch: &nested_branch
  start_at: parallel_step
  steps:
    parallel_step:
      type: parallel
      next: success
      branches:
        branch1: *simple_branch
        branch2: *simple_branch


# The pipeline of nested parallel branches
dag:
  start_at: parallel_step
  steps:
    parallel_step:
      type: parallel
      next: success
      branches:
        branch1: *nested_branch
        branch2: *nested_branch