Orchestration on Docker¶

APIs¶

dagster_docker.DockerRunLauncher RunLauncher[source]¶

Config Schema:

image (dagster.StringSource, optional):

The docker image to be used if the repository does not specify one.

network (dagster.StringSource, optional):

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional):

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource):

username (dagster.StringSource):

password (dagster.StringSource):

env_vars (List[String], optional):

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional):

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional):

Names of the networks to which to connect the launched container at creation time

Launches runs in a Docker container.

dagster_docker.docker_executor ExecutorDefinition[source]¶

Config Schema:

image (dagster.StringSource, optional):

The docker image to be used if the repository does not specify one.

network (dagster.StringSource, optional):

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional):

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource):

username (dagster.StringSource):

password (dagster.StringSource):

env_vars (List[String], optional):

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional):

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional):

Names of the networks to which to connect the launched container at creation time

retries (selector, optional):

Whether retries are enabled or not. By default, retries are enabled.

Default Value:

{
    "enabled": {}
}

Config Schema:

enabled (strict dict, optional):

Default Value:

{}

disabled (strict dict, optional):

Default Value:

{}

max_concurrent (dagster.IntSource, optional):

Limit on the number of containers that will run concurrently within the scope of a Dagster run. Note that this limit is per run, not global.

tag_concurrency_limits (List[strict dict], optional):

A set of limits that are applied to steps with particular tags. If a value is set, the limit is applied to only that key-value pair. If no value is set, the limit is applied across all values of that key. If the value is set to a dict with applyLimitPerUniqueValue: true, the limit will apply to the number of unique values for that key. Note that these limits are per run, not global.

Executor which launches steps as Docker containers.

To use the docker_executor, set it as the executor_def when defining a job:

from dagster_docker import docker_executor

from dagster import job

@job(executor_def=docker_executor)
def docker_job():
    pass

Then you can configure the executor with run config as follows:

execution:
  config:
    registry: ...
    network: ...
    networks: ...
    container_kwargs: ...

If you’re using the DockerRunLauncher, configuration set on the containers created by the run launcher will also be set on the containers that are created for each step.