Nothing in your dbt project tests what a metric means
Tests, contracts, and CI gates all stop at the pipeline. The semantic layer, where revenue and active user get defined, has no equivalent, and a passing suite proves nothing about the definition.
A dbt project can have full test coverage and still ship a wrong number. Not through a bug in a transformation, and not through a broken source. Through a definition that nobody validated, because no tool in the stack validates definitions.
I found this while evaluating AtScale. The modeling held up. What did not hold up was tracing an aggregated number back to the rows that produced it.
Where coverage stops
Tests, contracts, and CI gates cover sources, staging models, and marts. Above the marts sits the semantic layer, where "revenue" and "active user" get their meaning, and above that sit the dashboards and the AI answers people act on.
A unique test on a primary key proves the key is unique. It says nothing about whether the metric built on that key counts what its name claims. Those are different questions, and only one of them has tooling.
What it looks like
Consider a semantic layer reporting 42,300 active users for a period. The source table holds 40,500 distinct users. A join to a sessions table fans out, duplicating roughly 1,800 rows, and the aggregate counts the duplicates.
These figures are an illustration, not a measurement from a system I ran. The mechanism is real and common; the numbers are chosen to make the arithmetic legible.
Every base-table test passes. unique passes, because the source key is unique. not_null passes. Freshness passes. The relationship test passes, because the foreign key is valid. Nothing tests the join grain, and nothing tests what "active user" is supposed to mean.
A 4.3% overcount survives a green CI run.
Why the layer resists testing
Pipeline tests assert properties of rows. A metric definition is not a property of rows, it is an agreement about meaning, and the failure mode is a number that is internally consistent and externally wrong.
Three things make it worse:
The definition lives in more than one place. A metric in the semantic layer, the same metric hand-rolled in a dashboard, and a third version in a finance spreadsheet. All three are defended by whoever wrote them.
Aggregation hides the evidence. Once a number is a single figure on a tile, the rows behind it are gone from view. Row-level traceability is the thing I most often find missing, and its absence is what turns a definition question into an argument.
Ownership sits between two teams. The data engineer owns the pipeline that produces the table. The analyst owns the dashboard that reads it. The definition in between belongs to whoever asks about it last.
Three questions
Before trusting a metric enough to put an AI product on it:
Can I trace this number to its source rows? If the answer is no, you cannot debug it, and you cannot settle a disagreement about it either.
Who owns validating this definition, rather than this pipeline? A named person. If the answer is the data team, that means nobody.
What test would catch a wrong definition rather than a broken join? Usually this is a reconciliation against an independently produced figure, run on a schedule. It is unglamorous and it is the only thing that works.
The reason this matters more now than it did two years ago: a dashboard puts a wrong number in front of a person who might question it. A retrieval system puts it into a generated answer that reads as confident and carries no marker saying which figure was suspect.
Related reading
- Data platformA generated pipeline is a first draft, not autocompleteSnowflake CoCo builds a whole pipeline from one prompt. The code being easier to write is exactly why the review cannot get easier to skip.
- Data platformTest cadence is the missing dimension, not test countA freshness check proves the data looked right when CI ran. It proves nothing about the twenty-three hours afterwards, and AI-generated test suites write straight into that gap.
- Data platformBad data does not slow you down. It scales your mistakes.Teams invest in models and underinvest in the data feeding them, then wonder why outputs feel off. AI did not create the problem. It removed the human who used to catch it.