Data Platform Starter

A runnable baseline data platform: ingestion with dlt, a DuckDB warehouse (or MotherDuck), modelling with dbt and orchestration with Dagster. It comes with data contracts, quality gates and CI.

It's small enough to read in an afternoon and complete enough to clone as the start of a real platform. The example data product publishes daily ECB exchange rates per GBP.

uv sync
uv run invoke demo

Done. PASS=31 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=31
fx_rates_gbp: 660 rows, latest 2026-09-30. 1 GBP = EUR 1.1701, JPY 208.5932, USD 1.3286

The demo runs offline from recorded API responses, so it needs no keys or network access.

flowchart LR
    api["Frankfurter API<br/>(ECB rates)"] -->|dlt, incremental merge| raw
    subgraph warehouse["DuckDB / MotherDuck"]
        raw["raw_frankfurter<br/>as loaded"] --> stage["stage<br/>typed, renamed"]
        stage --> curate["curate<br/>facts and dimensions"]
        curate --> product["product<br/>contracted data products"]
    end
    product --> consumers["Reports, notebooks,<br/>AI workloads"]
    contract["ODCS contract"] -. describes .-> product
    dagster["Dagster<br/>daily schedule"] -. orchestrates .-> api
    dagster -. orchestrates .-> warehouse

Dagster sees dlt resources and dbt models as one asset graph, so lineage runs from the API to the product:

frankfurter/exchange_rates → stage/stg_frankfurter__exchange_rates → curate/fct_exchange_rates ┐
frankfurter/currencies     → stage/stg_frankfurter__currencies     → curate/dim_currencies      ┴→ product/fx_rates_gbp

Tests carry a severity. Errors block the build, because the data is wrong. Warnings alert someone, because the data might be wrong.

fx_rates_gbp is published under an Open Data Contract Standard contract in contracts/fx_rates_gbp.odcs.yaml. It covers the schema, the quality rules, the SLAs and the owner.

The same columns are enforced by dbt's model contract, so a build fails if the SQL produces anything else. tests/test_contract.py checks that the ODCS contract and the dbt contract describe the same table. That means the published promise and the enforced one can't drift apart.

Configuration is all environment variables:

Set DBT_SCHEMA_PREFIX to your branch or pull request, and every layer is built into its own schemas (pr_12_stage, pr_12_curate, pr_12_product). You can test changes against real data without touching anyone else's tables. Unset, models land in the plain layer schemas, the same in every environment. See dbt/macros/generate_schema_name.sql.

Every pull request runs lint (including SQL), the full test suite and a dependency audit. It also runs a secrets scan and a Docker smoke test that builds every model inside the image. The end-to-end tests run dlt and dbt against the recorded fixtures, so CI is fast, deterministic and needs no credentials.

• Swap the source. Replace sources.py with your own dlt source, record a few responses into fixtures/, and rename the staging models.
• Swap the warehouse. See docs/swapping-the-warehouse.md for Snowflake, BigQuery and Databricks.
• Add a product. Add a model under dbt/models/products/ with a contract, plus an ODCS contract in contracts/, and extend tests/test_contract.py.

Decisions and their reasoning are recorded in docs/adr/.

src/data_platform_starter/
    sources.py        dlt source: Frankfurter API, incremental
    pipeline.py       dlt pipeline and destination selection
    transform.py      dbt runner and schema naming
    definitions.py    Dagster assets, resources and schedule
    fixtures/         recorded API responses for offline runs
dbt/
    models/           staging/, curated/, products/
    tests/            generic and singular data tests
    macros/           branch-aware schema naming
contracts/            ODCS data contracts
tests/                unit, contract and end-to-end tests
docs/                 ADRs and guides

Generated from Qualixto/python-template, and kept up to date with uvx copier update --trust. Maintained by Qualixto, licensed Apache-2.0.