Test cadence is the missing dimension, not test count
A freshness check proves the data looked right when CI ran. It proves nothing about the twenty-three hours afterwards, and AI-generated test suites write straight into that gap.
Your dbt tests can pass every CI run and still miss the exact failure that reaches production.
A freshness or time-window test proves one thing: the data looked right at the moment CI ran. It says nothing about whether it stayed right.
Why it slips through
CI runs on its own schedule. Your pipeline breaks on its own schedule. Those two schedules do not have to line up, and usually do not.
Nobody gets paged, because nothing failed. The test passed. It passed at the only moment it was ever going to run.
This is different from the usual data quality complaint. Coverage is not the problem here. You can have a test on every column of every model and still carry this gap, because every one of those tests inherits the schedule of the run that invokes it.
Why AI makes it worse
A model will generate a full dbt test suite in minutes. Structurally valid, technically correct, easy to ship, and pointed at exactly the same blind spot. It writes tests that prove existence rather than continuous coverage, because that is what the examples in its training data look like.
dbt Labs put it plainly in their State of Analytics Engineering 2026 report: "AI-driven acceleration is outpacing trust and governance."
More tests, generated faster, is not more protection. It is the same coverage shape arriving sooner, with more surface area to maintain and a stronger feeling of safety.
The question to ask before shipping a test
Does this prove things looked fine when I checked, or does it run often enough to catch the failure that actually matters?
For most freshness and volume checks the honest answer is the first one. Fixing it is a scheduling decision rather than a testing decision:
Match the check interval to the tolerance, not to the CI run. If four hours of stale data is unacceptable, the check runs every hour, separately from CI. This usually means a scheduled job outside the deployment pipeline, which is why it rarely happens by default.
Alert on the absence of a run, not only on a failing one. A pipeline that stalls produces no failing test. It produces no test at all, and a monitor watching only for failures sees silence and reports health.
Separate deployment tests from operational monitors. They answer different questions. CI asks whether this change is safe to merge. A monitor asks whether the system is healthy right now. Using one for both leaves the second question unanswered on a daily cadence.
The teams I have seen get this right run far fewer tests than the teams that get it wrong. They run them on the clock the business actually cares about.
Related reading
- Data platformA generated pipeline is a first draft, not autocompleteSnowflake CoCo builds a whole pipeline from one prompt. The code being easier to write is exactly why the review cannot get easier to skip.
- Data platformBad data does not slow you down. It scales your mistakes.Teams invest in models and underinvest in the data feeding them, then wonder why outputs feel off. AI did not create the problem. It removed the human who used to catch it.
- Data platformNothing in your dbt project tests what a metric meansTests, contracts, and CI gates all stop at the pipeline. The semantic layer, where revenue and active user get defined, has no equivalent, and a passing suite proves nothing about the definition.