CI/CD in Dagster+ Serverless

note

This guide only applies to Dagster+ Serverless deployments. For Hybrid guidance, see CI/CD in Dagster+ Hybrid.

If you're a GitHub or GitLab user, you can use our predefined workflows to deploy and synchronize your code to Dagster+ Serverless. You can also use other Git providers, or a local Git repository with the dagster-cloud CLI to run your own CI/CD process.

note

Using the Connect to GitHub or Connect to GitLab apps in Dagster+ to configure a Git repository requires the Organization Admin role so that the system can provision an agent token.

GitHub

If you're a GitHub user, you can use our GitHub app with GitHub Actions to set up a repository containing basic code and configuration for you consistent with Dagster+ best practices.

Once you have set up the repo, pushing your code changes to the main branch will automatically deploy them to your prod Serverless full deployment. Pull requests will create ephemeral branch deployments that you can preview and test in the Dagster+ UI.

If you are importing a Dagster project in an existing GitHub repo

• The repo will need to allow the Workflow permission for Read and write permissions. Workflow permissions settings can be found in GitHub's Settings > Actions > General > Workflow permissions. In GitHub Enterprise, these permissions are controlled at the Organization level.

• An initial commit will need to be able to be merged directly to the repo's main branch to automatically add the GitHub Actions workflow files. If branch protection rules require changes be made through pull requests, this will prevent the automatic setup from completing. You can temporarily disable the branch protection rules and then re-enable them after the automatic setup completes.

GitLab

If you're a GitLab user, you can use our GitLab app to set up a repo containing basic code and CI/CD configuration for you consistent with Dagster+ best practices.

Once you have set up the repo, pushing your code changes to the main branch will automatically deploy them to your prod Serverless deployment. Merge requests will create ephemeral branch deployments that you can preview and test in the Dagster+ UI.

Other Git providers or local development

If you don't want to use our automated GitHub/GitLab process, you can use the dagster-cloud command-line CLI in another CI environment or locally.

1. First, create a new project by doing one of the following:

   • Cloning the Serverless quickstart example
   • Using the create-dagster project command. Note: If you create your project from the command line, you will need to add dagster-cloud as a dependency in your pyproject.toml file and add a dagster-plus-deploy.yml workflow file to the .github/workflows directory. See the Serverless quickstart example repo for examples of both.

2. Next, install the dagster-cloud CLI and use the configure command to authenticate it to your Dagster+ organization:

   pip install dagster-cloud
   dagster-cloud configure

You can also configure the dagster-cloud tool non-interactively; for more information, see the dagster-cloud installation and configuration docs.

3. Finally, deploy your project to Dagster+ using the serverless command:

dagster-cloud serverless deploy-python-executable ./my-project \
  --location-name example \
  --package-name quickstart_etl \
  --python-version 3.12

Windows variant

If you are using Windows, you will need to replace the deploy-python-executable command with deploy:

dagster-cloud serverless deploy ./my-project \
  --location-name example \
  --package-name quickstart_etl \
  --python-version 3.12