Dagster

Dagster treats data pipelines as software assets, not just DAGs. You define what data you produce, and Dagster tracks lineage, runs tests, and surfaces failures with full context.

What it does

Write pipelines in Python with built-in testing, type checking, and logging. The UI shows asset lineage (which tables/jobs depend on what), run history, and error details. Supports scheduling, sensors (trigger on file drops), and integrations with dbt, Airflow, Spark, and cloud storage.

Best for

• Data engineers building production-grade ETL/ELT pipelines
• Teams needing end-to-end observability across data workflows
• Organizations migrating from Airflow to a more asset-centric orchestrator

Strengths

• Asset-based lineage: see exactly which data assets are produced and consumed
• Built-in testing framework—validate pipeline logic before deployment
• UI shows run history, logs, and error stack traces in one view

• Asset-based pipelines — define data assets, not just tasks
• Lineage tracking — auto-generated dependency graph of all assets
• Testing framework — unit tests, integration tests, and type checks
• Scheduling & sensors — cron schedules and event-driven triggers
• dbt integration — run dbt models as Dagster assets
• Cloud storage — read/write from S3, GCS, Azure Blob
• UI dashboard — run history, logs, and error debugging