Configuring versioned state storage

Beta feature

This feature is considered in a beta stage. It is still being tested and may change. For more information, see the API lifecycle stages documentation.

When using the VERSIONED_STATE_STORAGE state management strategy, you need to configure a state storage backend in your Dagster instance. Under the hood, you are just configuring a UPath to your specific storage backend under which all state files will be stored.

Instance configuration

Configure the storage backend in your dagster.yaml file (or equivalent helm chart settings):

AWS S3

# dagster.yaml
defs_state_storage:
  module: dagster._core.storage.defs_state.blob_storage_state_storage
  class: UPathDefsStateStorage
  config:
    base_path: s3://my-bucket/dagster/defs-state
    storage_options:
      ...

Google Cloud Storage

# dagster.yaml
defs_state_storage:
  module: dagster._core.storage.defs_state.blob_storage_state_storage
  class: UPathDefsStateStorage
  config:
    base_path: gs://my-bucket/dagster/defs-state
    storage_options:
      ...

Azure Blob Storage

# dagster.yaml
defs_state_storage:
  module: dagster._core.storage.defs_state.blob_storage_state_storage
  class: UPathDefsStateStorage
  config:
    base_path: az://my-container/dagster/defs-state
    storage_options:
      ...

Refer to the universal-pathlib documentation for information on configuring your specific storage backend. The storage_options field is used to pass additional options to your specific UPath.

note

If you are deploying using Helm, make sure to set the includeInstance flag to true in your dagster-user-deployments values. This will mount the Dagster instance ConfigMap into your user code pods, enabling them to correctly load your definitions.

Next steps

• Learn how to manage state in CI/CD deployments