dbt-sentinel: measuring a review agent before believing it

A public build, done in the open over two weeks, with the measurements published whatever they said. A dbt project is a graph of SQL models: renaming a column in a staging model is one line in a diff and can break twelve dashboards downstream. dbt does not rewrite references, so the next run either fails or — worse — succeeds with wrong numbers.

The reviewer looking at that diff sees one line. Seeing the twelve consumers means walking the dependency graph by hand, so in practice nobody does it and breaking changes ship. The information needed to prevent this already sits in dbt's compiled manifest; nothing puts it in front of the reviewer at the moment they decide. But the harder problem is the one underneath: detecting a removed column is easy, and deciding whether removing it *matters* is not. That depends on who consumes it, whether a contract is enforced, and whether a join change altered the grain — two graph questions and one judgement call.

I wrote the evaluation set before the agent existed. Thirty fixtures — ten breaking, ten that should pass, ten subtle — were labelled from dbt semantics before the scoring code was written. Building the runner first produces a suite that confirms whatever the code already does; labelling first means the numbers can disprove me. Any label that later changes has to be argued in writing rather than silently edited.

I made deterministic code answer anything with an exact answer. The manifest ships a child map, so blast radius is a breadth-first traversal, not a retrieval problem. Embedding a dependency graph and asking a model to reason over it would be slower, costlier and less correct — reachability is exact, and an approximately-right blast radius is a confident lie. The model is spent on judgement, never on lookup.

I made reach amplify risk rather than create it. Nearly every staging model sits upstream of a dashboard, so the first scorer treated that proximity as risk and gave a comment-only edit a HIGH. That failure is behavioural, not statistical: a reviewer who sees HIGH on whitespace learns the badge is noise, and the next real HIGH gets ignored too. Severity is now triggered only by a structural change.

I stopped the model from writing the comment. It returns schema-validated finding objects and the rendering is template code. A model that emits markdown can forge a heading or a severity badge inside a comment carrying the project's name — and precision cannot be computed over prose, so the eval suite depends on that structure existing.

I made every failure path degrade visibly. No API key, a timeout, a rate limit, an invalid schema twice, prose instead of a tool call — each falls back to deterministic-only output with the reason printed in the comment. A reader who cannot tell the agent failed will read 'no findings' as 'nothing to worry about' when it actually means 'the API timed out'.

I published the numbers that make it look worse. Recall is 0.533 against a 0.70 target and the false-positive rate is 0.200 against a 0.10 target — two of six targets missed, two more unmeasured. That is the honest shape of a structural-only scorer: when it fires it is usually right, and it misses more than half of what a reviewer should catch. Every miss lives in SQL semantics that graph traversal cannot see. Publishing the gap is what makes the case for the agent layer credible instead of retrofitted.

I deliberately did not fix the two worst failure modes yet. Four fixtures resolve to zero nodes and are silently dropped, two of them HIGH — the worst failure shape in the project. Additive columns score as breaking, which drives the false-positive rate. Both are diagnosed down to the specific predicate and parked, because fixing them in the same session that measured them destroys the before-and-after comparison that makes the fix demonstrable.

I report model-level lineage, not column-level. The tool says twelve models are downstream, not which of them actually select the dropped column. Column-level lineage needs a real SQL parser rather than the regex extraction it uses, and I put that on the non-goals list before writing any code rather than discovering it as scope creep halfway through.

There is no agent number in the README because there is no agent number. The judgement layer is built and the comparison against the deterministic baseline has not been run. Writing an estimate would have been easy and would have quietly turned a measured project into an unmeasured one.

Precision 0.800 and retrieval precision@3 0.708, both above target, measured against a real manifest compiled by dbt rather than a synthetic fixture.

The eval suite runs in CI as a regression gate: if a published metric drops below its recorded value the build fails, so a change that quietly makes review quality worse cannot merge while the README still claims otherwise.

Measuring first caught a bug that made the tool a silent no-op — a manifest compiled on Windows stores backslash paths while diffs use forward slashes, so nothing resolved and a breaking PR passed clean. The synthetic fixture had hidden it completely.

It also caught a retrieval defect that took precision@3 from 0.400 to 0.708, a cost constant that would have inflated every published figure by roughly half, and three of my own tests that were passing for the wrong reason.

Not yet deployed. The GitHub App is wired and tested against a fake client; no real pull request has received a comment, and the README says so.