Practice: Data pipeline orchestration with Airflow and Dagster
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: Dagster · Dagster makes the asset model concrete on a laptop with one local process and UI, so the learner can practise dependencies, backfills, schedules and operations and contrast each with the task model Airflow uses. · Alternatives: Apache Airflow models a pipeline as a DAG of tasks and is what most existing stacks already run; Prefect runs plain Python flows with little ceremony; dbt orders only SQL transformations inside a warehouse; Celery is a task queue with no data lineage, where Dagster is built around data assets and their lineage.
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Confirm the prerequisites for the first pipeline tutorial: a supported Python version and the project tooling the docs list.
Every later step assumes the scaffolding command line tool works, and a version mismatch is the most common first failure.
Explained in: Build your first Dagster pipeline | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The Python and tooling version checks named in the section print versions that meet the stated minimums.
2. Scaffold a new Dagster project in an empty folder and create its virtual environment.
The scaffold gives the standard layout and a pinned dependency file, which is what makes the work reproducible later.
Explained in: Projects | Dagster Docs (dagster.io), checked 2026-10-11
Done when: A project folder exists with a source package, a definitions module and a dependency file.
3. Start the Dagster webserver from the project root and open the local UI in a browser.
The UI is where you watch runs, lineage and automation for the rest of the path, and it is the local stand-in for a deployed instance.
Explained in: Projects | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The local UI loads in the browser and shows the project as a code location with no load errors.
Steps
1. Create an assets file and define one asset that reads the tutorial sample data and writes a table.
An asset is the unit Dagster orchestrates, so the first thing to learn is that you declare the data you want, not the task that produces it.
Explained in: Build your first Dagster pipeline | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The asset appears by name in the UI asset list.
2. Run the pipeline twice in a row and verify the output is identical after the second run.
An idempotent asset gives the same result however many times it runs, which is what makes retries and backfills safe. If the second run duplicates rows, rewrite the asset to overwrite its output instead of appending.
Explained in: Build your first Dagster pipeline | Dagster Docs (dagster.io), checked 2026-10-11
Done when: Two successful runs show in the UI and the row count of the output is the same after each.
3. Write a one-page map of Airflow architecture onto Dagster concepts: DAG to job, task and operator to op, scheduler to schedule and sensor, executor to executor.
Most customers already run Airflow, so you need to translate its scheduler, executor, DAG and task vocabulary into the Dagster object model and back. The concepts page defines each Dagster term you are mapping to.
Explained in: Concepts | Dagster Docs (dagster.io), checked 2026-10-11
Done when: You have a table that pairs each Airflow component with a Dagster concept and notes one difference for each pair.
4. Create a downstream asset that depends on the first one and materialize both from the UI.
In the asset model the dependency graph comes from which data each asset reads, not from hand-wired task ordering. Seeing the lineage graph is the fastest way to grasp that.
Explained in: Asset dependencies | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The lineage view shows an edge between the two assets and both show a successful materialization.
5. Write the same extract, transform and load flow as three separate assets, then compare it with the single asset version.
How finely you split assets decides what you can rerun, observe and backfill on its own. This is the central design choice of the asset model.
Explained in: Modeling ETL pipelines with assets | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The UI shows three linked assets and you can materialize only the last one without rerunning the first two.
6. Write an asset check that fails when the output table is empty or has null keys, then run it.
Operating a pipeline means knowing when data is wrong, not only when code crashed. Checks attach that test to the asset itself.
Explained in: Asset checks | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The check shows as passed on the asset page, and shows as failed after you deliberately break the input data.
7. Create a daily partitioned version of an asset and launch a backfill over the last seven partitions from the UI.
A backfill reruns history one partition at a time, and it is only safe because each partition run is idempotent. This is the Dagster counterpart of an Airflow backfill over past schedule intervals.
Explained in: Partitions and backfills | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The asset page shows seven materialized partitions, and rerunning one of them leaves its output unchanged.
8. Compare your project against the documented project structure and register every asset, check and schedule in the definitions the code location loads.
A code location is the packaged unit that Dagster loads and that you promote between environments. Anything not registered there does not exist for the orchestrator.
Explained in: Projects | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The definitions check command reports that all definitions load, and the layout matches the documented structure.
9. Enable a schedule that materializes your assets every day and turn it on in the UI.
Scheduling is the core job of an orchestrator, and it is off until you enable it. Many first deployments silently never run for that reason.
Explained in: Automation | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The schedule shows as running in the automation view with a next tick time.
10. Configure a time window freshness policy on the final asset so that it is flagged when it has not updated within a day.
Freshness policies turn an expectation about when the data should arrive into something the system tracks. That is how you operate a pipeline by its outputs and not its task logs.
Explained in: Asset freshness policies | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The asset page shows a freshness status, and it changes to a failing state once the window passes with no new materialization.
11. Write a half-page decision note saying when you would extend a customer's existing task-based orchestrator and when you would build with assets.
A forward deployed engineer usually extends what the customer already runs, so the choice must rest on stated criteria such as lineage needs, team skills and operating cost. The assets versus ops section gives the trade-off in Dagster terms.
Explained in: Assets | Dagster Docs (dagster.io), checked 2026-10-11
Done when: The note names at least three criteria and gives a clear recommendation for one concrete customer scenario.
12. Deploy the project following the production step; the hosted option is a paid cloud, so the closest local substitute is to run the project from a clean clone in a freshly created pinned environment with the schedule enabled.
A production-like run proves the project works from its pinned dependencies and registered definitions alone, with nothing left over from your working folder. That is the real test before handing it to a customer.
Explained in: Build your first Dagster pipeline | Dagster Docs (dagster.io), checked 2026-10-11
Done when: From a clean clone, the UI loads the code location without errors and a scheduled run completes with all checks passing.
Back to Data pipeline orchestration with Airflow and Dagster