An agentic QA system that reviews data changes before they ship

I led a team of 6–7 data quality analysts reviewing proposed changes to a large analytics platform. Reviewing a change meant an analyst reading the affected tables, working out what should be true about them, hand-writing the checks, running them, and assembling the evidence — the same reasoning re-applied to every new change. I designed the system, wrote its behavioural contracts, and shipped it into the team's actual review workflow.

Review quality depended on which analyst picked up the ticket and how much time they had. A thorough review took a day or more, so coverage was a function of available analyst hours rather than of what the change actually warranted. Two failure modes followed: shallow reviews let bad data through, and inconsistent reviews meant two analysts could reach different conclusions on the same change with no way to tell which was right. It was also the least interesting part of the job, which is a retention problem as much as a throughput one.

I split the system by whether a step has a testable correct answer. Parsing the intake, validating contracts, orchestrating the run, checking primary keys and completeness, enforcing SQL safety, and compiling evidence are all deterministic Python — they must produce the same result every run, so a model has no business in them. The model is spent only where correctness is a judgement call rather than a specification.

I gave the judgement dimensions bounded workers rather than free rein. Each one writes a scoped, read-only query plan, executes it only through a supplied query tool, and returns condition assessments with evidence references. Findings are warning-only by design: the system reports what looks wrong and why, and never converts that into a pass or fail the reviewer did not make.

I made the operator surface one action. The analyst fills an intake sheet naming the tables, keys, filters and requested dimensions, then invokes a single command with its path. Run ID generation, conversion, orchestration, worker dispatch, and report compilation all happen internally — there is no sequence of manual steps to get wrong.

I designed the evidence rules around privacy rather than bolting them on. Identifiers, names, addresses, contact details and free text are evidenced only as aggregates, formats, lengths, counts or masked values; reports carry assessments and query references, never raw sample rows. That constraint is what let the system read production data at all.

I made every run auditable and comparable. Each produces a machine-readable rollup, a compiled SQL artifact of all committed evidence, and a reviewer report that compares against the newest earlier run for the same scope — comparing only deterministic counts, statuses and freshness values, never model prose or timing, so a baseline can never be set by something that varies between runs.

The system never updates the pull request status. It is advisory: it assembles the evidence and states what it found, and a human reviewer owns the merge decision. Gating merges automatically was within reach and would demo far better, but this is the system that decides whether the business trusts its own data. A wrong automated block trains people to override it, and a wrong automated pass is worse — it launders bad data through a process that looks rigorous. I chose the version the team would keep relying on over the version that looks more autonomous.

The judgement dimensions return warnings rather than verdicts, and one of them returns only 'observed' or 'warn' with no failure state at all, because the underlying question has no freshness threshold the business had actually agreed. Inventing one would have produced confident output about a standard nobody set. I would rather the system say less and be right than manufacture a number to fill a column.

I kept the model out of steps it could plausibly have handled. Formatting, retries, validation and the final write are all specifications, so they are code — cheaper, repeatable, and testable in CI without any model access at all. The unit suite runs credential-free against deterministic mock evidence, which is only possible because those paths never call a model.

A review that took a day or more now runs in under an hour, including the analyst review step that gates it.

It runs in production as part of how the team reviews changes — the measure I care about most, because an internal tool that does not get adopted is just a demo.

Review output became consistent across analysts: the same change now produces the same evidence and the same rollup regardless of who ran it, which was not true before.

Mock and live runs are labelled distinctly in every report, so evidence from a test run can never be mistaken for evidence from a real one.