Data Orchestration Fundamentals
Understand what data orchestration is, why it matters, and how Dagster approaches it differently than competitors.
Why this matters
Every conversation you have at Dagster — internal, with AEs, with prospects — assumes orchestration vocabulary. If you do not know what an asset is or why someone would pick Dagster over Airflow, you cannot score accounts, write enrichment prompts, or evaluate fit signals. This is the load-bearing module. Everything else gets easier once this clicks.
Key concepts
- Data pipeline: a sequence of steps that move and transform data (e.g., Fivetran pulls from Salesforce → dbt models it in Snowflake → Looker reads it).
- Why pipelines fail without orchestration: no shared schedule, no dependency awareness, silent failures, no lineage when a number is wrong, manual reruns, slow debugging.
- DAG (Directed Acyclic Graph): the dependency map of a pipeline. "A must run before B, B before C, no loops."
- Asset: a data product that exists in storage (a table, a file, a model). Dagster's core unit.
- Job: a runnable unit that materializes one or more assets.
- Sensor: a trigger that runs a job when something happens (file lands, time passes, upstream finishes).
- Partition: a slice of an asset (e.g., one day of events). Lets you rerun just the broken slice.
- IO Manager: the code that handles reading and writing an asset to storage. Decouples logic from storage.
- Materialization: the act of computing an asset and writing it.
- Lineage: the graph of which assets depend on which. Click an asset, see what feeds it and what consumes it.
- Workflow-oriented vs asset-oriented: Airflow asks "what tasks should run?" Dagster asks "what data products should exist?" The asset-first model is the core philosophical difference.
- Where Dagster sits: ingestion (Fivetran/Airbyte) → storage (Snowflake/Databricks/BigQuery) → transformation (dbt) → orchestration (Dagster) → BI/AI (Looker, Hex, ML models).
Reading and watching
- Dagster University — Essentials intro path (sign up free, complete in browser)
- What Assets Do Best — animated explainer (5 min)
- Dagster Blog — find the "Software-Defined Assets" post (Nick Schrock author), read in full
- docs.dagster.io — do not read end to end. Skim the table of contents only. Bookmark for reference.
Hands-on exercises
Complete Dagster University Essentials
Auto-savedWork through the full intro course in the browser. Materialize at least one asset. Then write two sentences explaining what you just did to someone who has never heard of Dagster. No jargon. Imagine you are explaining it to your mom.
Self-check
Answer from memory first. Then reveal the model answer and grade yourself honestly.
In your own words, what is the difference between an asset and a job?
A pipeline runs every night. The Tuesday run produced wrong numbers in a downstream report. How does Dagster help you find the cause faster than a cron-based setup would?
What is a sensor and give one example of when you would use one?
Why is asset-oriented considered a different paradigm from workflow-oriented?
Where does Dagster sit in the modern data stack and what tools does it typically coordinate?