Skip to main content

Converting an existing project to use dg

info

This feature is considered in a preview stage and is under active development. There may be API changes and feature gaps. Please go to the #dg-components channel in our Slack to report issues or give feedback.

Suppose we have an existing Dagster project. Our project defines a Python package with a a single Dagster asset. The asset is exposed in a top-level Definitions object in my_existing_project/definitions.py. We'll consider both a case where we have been using uv with pyproject.toml and pip with setup.py.

tree
.
├── my_existing_project
│   ├── __init__.py
│   ├── assets.py
│   ├── definitions.py
│   └── py.typed
├── pyproject.toml
└── uv.lock

2 directories, 6 files

dg needs to be able to resolve a Python environment for your project. This environment must include an installation of your project package. By default, a project's environment will resolve to whatever virtual environment is currently activated in the shell, or system Python if no virtual environment is activated.

Before proceeding, we'll make sure we have an activated and up-to-date virtual environment in the project root. Having the virtual environment located in the project root is recommended (particularly when using uv) but not required.

If you don't have a virtual environment yet, run:

uv sync

Then activate it:

source .venv/bin/activate

Install dependencies​

Install the dg command line tool​

First let's install dg globally as a uv tool:

uv tool install dagster-dg

This installs dg into a hidden, isolated Python environment separate from your project virtual environment. The dg executable is always available in the user's $PATH, regardless of any virtual environment activation in the shell. This is the recommended way to work with dg if you are using uv.

Update project structure​

Add dg configuration​

The dg command recognizes Dagster projects through the presence of TOML configuration. This may be either a pyproject.toml file with a tool.dg section or a dg.toml file. Let's add this configuration:

Since our project already has a pyproject.toml file, we can just add the requisite tool.dg section to the file:

pyproject.toml
...
[tool.dg]
directory_type = "project"

[tool.dg.project]
root_module = "my_existing_project"
code_location_target_module = "my_existing_project.definitions"

There are three settings:

  • directory_type = "project": This is how dg identifies your package as a Dagster project. This is required.
  • project.root_module = "my_existing_project": This points to the root module of your project. This is also required.
  • project.code_location_target_module = "my_existing_project.definitions": This tells dg where to find the top-level Definitions object in your project. This actually defaults to [root_module].definitions, so it is not strictly necessary for us to set it here, but we are including this setting in order to be explicit--existing projects might have the top-level Definitions object defined in a different module, in which case this setting is required.

Now that these settings are in place, you can interact with your project using dg. If we run dg list defs we can see the sole existing asset in our project:

dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                         ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key      ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ my_asset │ default │      │       │             │ │
│         │ └──────────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴─────────────────────────────────────────────────────┘

Create a defs directory​

Part of the dg experience is autoloading definitions. This means automatically picking up any definitions that exist in a particular module. We are going to create a new submodule named my_existing_project.defs (defs is the conventional name of the module for where definitions live in dg) from which we will autoload definitions.

mkdir my_existing_project/defs

Modify top-level definitions​

Autoloading is provided by a function that returns a Definitions object. Because we already have some other definitions in our project, we'll combine those with the autoloaded ones from my_existing_project.defs.

To do so, you'll need to modify your definitions.py file, or whichever file contains your top-level Definitions object.

You'll autoload definitions using load_defs, then merge them with your existing definitions using Definitions.merge. You pass load_defs the defs module you just created:

import dagster as dg
from my_existing_project.assets import my_asset

defs = dg.Definitions(
    assets=[my_asset],
)

Now let's add an asset to the new defs module. Create my_existing_project/defs/autoloaded_asset.py with the following contents:

import dagster as dg


@dg.asset
def autoloaded_asset(): ...

Finally, let's confirm the new asset is being autoloaded. Run dg list defs again and you should see both the new autoloaded_asset and old my_asset:

dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                                 ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key              ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ autoloaded_asset │ default │      │       │             │ │
│         │ ├──────────────────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ my_asset         │ default │      │       │             │ │
│         │ └──────────────────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴─────────────────────────────────────────────────────────────┘

Now your project is fully compatible with dg!

Next steps​