Configuring and running a Docker agent#

This guide is applicable to Dagster Cloud.

In this guide, you'll configure and run a Docker agent. Docker agents are used to launch your code in Docker containers.

Prerequisites#

To complete the steps in this guide, you'll need:

  • Permissions in Dagster Cloud that allow you to manage agent tokens. Refer to the User permissions documentation for more info.

  • To have Docker installed

  • Access to a container registry to which you can push images with Dagster code. Additionally, your Docker agent must have the permissions required to pull images from the registry.

    This can be:

Step 1: Generate a Dagster Cloud agent token#

In this step, you'll generate a token for the Dagster Cloud agent. The Dagster Cloud agent will use this to authenticate to the agent API.

  1. Sign in to your Dagster Cloud instance.
  2. Click the user menu (your icon) > Cloud settings.
  3. In the Cloud settings page, click the Tokens tab.
  4. Click the + Create agent token button.
  5. After the token has been created, click Reveal token.

Keep the token somewhere handy - you'll need it to complete the setup.

Step 2: Create a Docker agent#

  1. Create a Docker network for your agent:

    docker network create dagster_cloud_agent

  2. Create a dagster.yaml file:

    instance_class:
  module: dagster_cloud.instance
  class: DagsterCloudAgentInstance

dagster_cloud_api:
  agent_token: <YOUR_AGENT_TOKEN>
  branch_deployments: true # enables branch deployments
  deployment: prod

user_code_launcher:
  module: dagster_cloud.workspace.docker
  class: DockerUserCodeLauncher
  config:
    networks:
      - dagster_cloud_agent

  3. In the file, fill in the following:

    • agent_token - Add the agent token you created in Step 1

    • deployment - Enter the deployment associated with this instance of the agent.

      In the above example, we specified prod as the deployment. This is present when Dagster Cloud organizations are first created.

  4. Save the file.

Step 3: Start the agent#

Next, you'll start the agent as a container. Run the following command in the same folder as your dagster.yaml file:

docker run \
  --network=dagster_cloud_agent \
  --volume $PWD/dagster.yaml:/opt/dagster/app/dagster.yaml:ro \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --restart on-failure \
  -it docker.io/dagster/dagster-cloud-agent:latest \
  dagster-cloud agent run /opt/dagster/app

This command:

  • Starts the agent with your local dagster.yaml mounted as a volume
  • Starts the system Docker socket mounted as a volume, allowing the agent to launch containers. The --restart-on-failure argument will ensure that if it fails, Docker will restart it.

To view the agent in Dagster Cloud, navigate to the Status page and click the Agents tab. You should see the agent running in the Agent statuses section:

Instance Status

Credential Helpers#

If your images are stored in a private registry, configuring a Docker credentials helper allows the agent to log in to your registry. The agent image comes with several popular credentials helpers preinstalled:

These credential helpers generally are configured in ~/.docker.config.json. To use one, make sure you mount that file as a volume when you start your agent:

  docker run \
    --network=dagster_cloud_agent \
    --volume $PWD/dagster.yaml:/opt/dagster/app/dagster.yaml:ro \
    --volume /var/run/docker.sock:/var/run/docker.sock \
    --volume ~/.docker/config.json:/root/.docker/config.json:ro \
    --restart on-failure \
    -it docker.io/dagster/dagster-cloud-agent:latest \
    dagster-cloud agent run /opt/dagster/app

Next steps#

Now that you've got your agent running, what's next?