Dagster

Dagster is an open-source data orchestrator built around software-defined assets: instead of describing a pipeline as a sequence of tasks, you declare the tables, files, and ML models it should produce and the code that computes each one, and Dagster derives the dependency graph from there. Pipelines are plain Python, so they can be tested and run locally before they are deployed.

The web UI shows the asset graph with lineage, run history, and each asset's freshness, while schedules, sensors, and declarative automation decide when assets are rebuilt. Integrations cover dbt (each dbt model becomes a Dagster asset), Snowflake, Databricks, Spark, Airbyte, Fivetran, DuckDB, and the major clouds, so Dagster often orchestrates tools a team already runs rather than replacing them.

The core is Apache 2.0 licensed and self-hostable on Docker or Kubernetes. Dagster+ is the managed service from Dagster Labs, with serverless or hybrid deployment (the hybrid agent runs your code in your own infrastructure) and plans billed in credits, where each asset materialization or op execution counts as one. It suits data teams that want lineage and data quality built into orchestration, and it is most often compared with Airflow and Prefect.