concept · concepts/Pipeline Legibility.md
Pipeline Legibility
Design a pipeline so the next reader can tell what's actually happening from what they can see — name objects for the direction data flows, trust only the lineage you can read, and build ingestion to the source's real export contract. The opposite is a pipeline whose names and assumptions quietly lie.
Explanation
The source (Working Heuristics — Data, Matching, Product, Org, Tooling) bundles three rules that all serve one goal — a pipeline that doesn't deceive whoever reads it next:
- Name tables for the direction data flows, or you encode a lie that misleads the next reader. A table named as if data flows one way while it actually flows the other is worse than an unnamed one: it actively sends people wrong.
- Trust lineage you can read (DDL / source); don't assert the connector tech you can't see. Claim only what the artifacts in front of you actually show. If you can read the DDL, trust it; if the ingestion connector is opaque, don't narrate a mechanism you're guessing at.
- Ingestion design is downstream of the source's export contract. Confirm how the source actually exports (full vs. incremental, push vs. pull, file vs. API, ordering and dedup guarantees) before committing to a pipeline. The export contract is the upstream fact; your ingestion is contingent on it.
Why it matters / where it shows up
- Legibility is a gift to the future reader (often yourself). Honest naming and readable lineage are how a pipeline stays debuggable as it ages and changes hands.
- It prevents the most expensive class of mistake — building a whole ingestion on an assumed export behavior, then discovering the source exports differently.
- Directionality is load-bearing across domains. "Name tables for the direction data flows" is the data-layer twin of "[[Scope Hygiene for Tickets and Tooling|directional links are load-bearing]]" — a reversed relationship gets silently misfiled in both a data model and a ticket tracker.
Connections
- Lands vs Lives — locating where data physically lands vs. where it's modeled is the other half of reading a pipeline honestly.
- Medallion Architecture — clear per-layer naming is what keeps a multi-hop pipeline legible.
- Data Contract — a contract is only as trustworthy as the lineage feeding it.
- System of Record — "trust lineage you can read" is the evidentiary version of "verify authority before promoting a source to SOT."
- Scope Hygiene for Tickets and Tooling — shares the "directionality is load-bearing" principle in the tooling domain.
Sources
- Working Heuristics — Data, Matching, Product, Org, Tooling — name for flow direction; trust readable lineage; ingestion is downstream of the export contract.