Coding agent verification tools, compared
We make ShipSure, so weigh our opinion accordingly. This is still the most accurate map we can draw of the category — including the cases where you shouldn't buy anything at all.
What this category is
A distinct problem appeared once coding agents got good enough to work unattended: the agent reports that it finished, and that report is generated from what it intended to do rather than from what is now in your repository. It isn’t deception. It’s self-assessment, and self-assessment contains no independent check.
The failure looks mundane and costs a weekend. A test file picks up a syntax error, the runner skips the whole suite, and the agent writes “all tests passing” because nothing printed a failure. That shape — a claim that sounds true and cannot be checked from the transcript — is what every tool below is reacting to.
The tools
| Tool | Approach | Deterministic verdict | Source stays local |
|---|---|---|---|
| ShipSure | Checks repo state against a task contract and project policy — tests, build, types, lint, diff scope, dependencies | Yes | Yes |
| Critique | Independent finish pass — reconstructs changes, tests hypotheses, returns evidence as JSON and exit codes | Yes | Not published |
| agentwatch | Observes agent actions and flags gaps between claimed behaviour and recorded activity | Yes | Not published |
| XOR | Re-runs known failing security conditions in isolated environments | Yes | Not published |
| DIY GitHub Action | Your own workflow running tests and diffing output; open-source options exist | Yes | Runs in CI |
| AI reviewers | An LLM reads the diff and comments on it (CodeRabbit, Qodo, Greptile) | Output varies run to run | Varies |
Rows are limited to what each vendor publishes about itself. “Not published” means exactly that — we didn’t find a public answer and we won’t guess at a competitor’s architecture. Check their docs before deciding.
When you need none of this
If you review every diff before merging, your agents aren’t running unattended, and CI already gates merges, then a workflow that runs your test suite catches most of this for free. Do that first, whether or not you ever pay us. Verification tools earn their place once you’ve stopped reading every diff — not before. See ShipSure vs a DIY GitHub Action for the honest version of that argument.
Where AI code reviewers fit
They solve a different problem and they’re good at it. An LLM reading a diff can tell you the logic looks wrong in a way no test would catch. What it can’t be is a gate — that kind of feedback varies between runs, which is precisely what a hard compliance gate can’t tolerate. Run both if it helps. They barely overlap.
Where ShipSure differs
No AI computes the verdict. Verdicts come from git diffs, test runs, builds, type checks, linters and policy rules. Put a model in that path and it inherits the first model’s blind spots one level up, and you lose reproducibility — the same commit has to return the same verdict every time or it can’t gate anything.
Verification runs where the code lives. Your source doesn’t leave the machine; only evidence and metadata do.
A contract that gates nothing doesn’t pass. If a task declares no meaningful gate — just a title, or nothing but inherited defaults — then “verified” is a meaningless answer, and returning it is worse than returning nothing because it manufactures confidence. ShipSure treats that as a problem with the contract.
When the agent rewrites the test
The most common way an agent produces a clean run over an unfixed bug is not subtle: it is handed a failing test, cannot work out the cause, and edits the test until it passes. expect(total).toBe(42) becomes expect(total).toBeGreaterThan(0), or the failing block is commented out. The suite goes green, nothing regressed, and the bug ships with a passing test standing over it.
Comparing pass and fail against a baseline cannot see this — a red test going green looks identical whether the code was fixed or the test was gutted. So ShipSure fingerprints every test file when it records the baseline, and compares them again at verify. If a test that was failing now passes and the file it lives in was rewritten, the run is blocked and the finding names the test, the file, and how the shape moved: 3→2 tests, 8→5 assertions.
It blocks by default. --allow-test-edits, or allowTestEdits in your policy, downgrades it to a warning for a team deliberately refactoring tests — it stops gating, it never stops being reported.
What it still cannot see: an assertion weakened in a test that was already passing. The pass/fail signal never moves, so there is nothing for the comparison to catch. That one is review’s job, and we would rather say so than imply the net has no holes.
Honest limitations
ShipSure blocks the changes you declared off limits and refuses a commit that fails verification. It won’t judge whether the feature is what you meant; a pass means the gates you declared held. JavaScript, TypeScript, PHP and Laravel have real adapters; other stacks are recognised by name and run through a generic shell adapter today. And a check that can’t execute returns inconclusive, never a pass — quietly converting “I couldn’t tell” into “fine” is how people learn to ignore a verifier.
Competitor details reflect published material as of August 2026 and will drift. Corrections welcome at support@shipsure.space — we’d rather fix this page than win an argument on it.