Skip to content

local-container

Execute all the steps of the pipeline in containers.

  • Provides a way to test the containers and the execution of the pipeline in local environment.
  • Any failure in cloud native container environments can be replicated in local environments.
  • Ability to provide specialized compute environments for different steps of the pipeline.
  • The scalability is still constrained by the resources in local environment.

Configuration

In the mode of local-container, we execute all the commands in a container.

Ensure that the local compute has enough resources to finish all your jobs.

Configuration options:

pipeline-executor:
  type: local-container
  config:
    docker_image: <required>
    auto_remove_container: true/false
    environment:
      key: value
    overrides:
      alternate_config:
        docker_image: <required>
        auto_remove_container: true/false
        environment:
          key: value
  • docker_image: The default docker image to use for all the steps.
  • auto_remove_container: Remove container after execution
  • environment: Environment variables to pass to the container

Overrides give you the ability to override the default docker image for a single step. A step can then then refer to the alternate_config in the task definition.

Example:

from runnable import PythonTask

task = PythonTask(
    name="alt_task",
    overrides={
        "local-container": "alternate_config"
        }
    )

In the above example, alt_task will run in the docker image/configuration as defined in the alternate_config.

runnable does not build the docker image for you, it is still left for the user to build and ensure that the docker image provided is the correct one.

Debugging

auto_remove_container allows you to run the failed container independently to identify the issue that caused the failure.

All the examples in the concepts section can be executed using the below configuration:

pipeline-executor:
  type: local-container
  config:
    docker_image: <your docker image>

Dynamic docker image

The docker image can provided at the run time by using environmental variables.

For example:

pipeline-executor:
  type: local-container
  config:
    docker_image: $docker_image

The $docker_image will be replaced by the environmental variable RUNNABLE_VAR_docker_image during run time. The same rule applies to overrides too.