Dagster daemon#

Several Dagster features, like schedules, sensors, and run queueing, require a long-running dagster-daemon process to be included with your deployment.

Starting the daemon#

Running locally
Deploying the daemon

Running locally#

The Dagster daemon can be started locally in a few ways, which are outlined in the following tabs. Once started, the process should be kept running.

The Dagster daemon can load a file directly as a code location. In the following example, we used the -f argument to supply the name of the file to dagster-daemon:


dagster-daemon run -f my_file.py

This command loads the definitions in my_file.py as a code location in the same Python environment where the daemon resides.

You can also include multiple files at a time:


dagster-daemon run -f my_file.py -f my_second_file.py

The Dagster daemon can also load Python modules as code locations. When this approach is used, Dagster loads the definitions defined at the top-level of the module, in a variable containing the Definitions object of its root __init__.py file. As this style of development eliminates an entire class of Python import errors, we strongly recommend it for Dagster projects deployed to production.

In the following example, we used the -m argument to supply the name of the module to the daemon process:


dagster-daemon run -m your_module_name

This command loads the definitions in the variable containing the Definitions object in the named module - defined as the root __init__.py file - in the same virtual environment as the daemon.

To load definitions without supplying command line arguments, you can use the pyproject.toml file. This file, included in all Dagster example projects, contains a tool.dagster section with a module_name variable:


[tool.dagster]
module_name = "your_module_name"  ## name of project's Python module

When defined, you can run this in the same directory as the pyproject.toml file:


dagster-daemon run

Instead of this:


dagster-daemon run -m your_module_name

Deploying the daemon#

For information on deploying the daemon to environments like Docker or Kubernetes, refer to the deployment guides.

Available daemons#

The dagster-daemon process reads from your Dagster instance file to determine which daemons should be included. Each of the included daemons then runs on a regular interval in its own threads.

The following daemons are currently available:

Name	Description	Enabled by
Scheduler daemon	Creates runs from active schedules	Enabled / runs as long as the default `DagsterDaemonScheduler` isn't overriden as the scheduler on your instance.
Run queue daemon	Launches queued runs, taking into account any limits and prioritization rules set on your instance	Setting the run coordinator on your instance to `QueuedRunCoordinator`.
Sensor daemon	Creates runs from active sensors that are turned on	Always enabled
Run monitoring daemon	Handles run worker failures	Using the `run_monitoring` field in your instance. Refer to the Run monitoring documentation for more info.

If the daemon is configured to use a workspace file to load code location(s), note that they will periodically reload the file. This means that the dagster-daemon process doesn't need to be restarted when workspace files are changed.

Checking daemon status in Dagit#

To check the status of the dagster-daemon process within Dagit:

In the top navigation, click Deployment.
Click the Daemons tab.

This tab displays information about all the daemons currently configured on your instance.

Each daemon periodically writes a heartbeat to your instance storage. If a daemon doesn't show a recent heartbeat, check the logs from your dagster-daemon process for errors.