Open source · MIT license · runs in your CI
Groundtruth reads the whole changed function, checks its real callers, and never posts a finding unless it can quote the exact line and survive a second, skeptical model. No vector database, no hosted service, no seat license — your own key, any provider, running in the CI you already have.
New promos argument is required, but checkout.py still calls calculate_discount(price).
groundtruth eval --cases cases replays it offlineSeven stages, each handing the next exactly one thing. Only three call a model at all, and each of those is a bounded, isolated round trip — everything else is deterministic code: parsing, searching, quote matching, fingerprinting.
The handoff, stage by stage
A pull request event starts the run. The CI job checks out the repository with full history and calls one command, groundtruth review.
Why CI: the platform's own runner is already the queue and the compute. Nothing has to be hosted, so installing the tool is one workflow file and one secret.
Git provides the diff and the full text of each changed file at both the base and the head commit.
Why both versions: the diff alone shows changed lines without the code around them, and comparing the two versions is what reveals that a function's signature changed.
This stage decides what the model is allowed to see. It makes no AI calls.
Why an AST instead of a vector database: parsing is exact and repeatable. Similarity search returns code that merely looks related, and a wrong snippet produces a confident, wrong finding. This way, context is always real code that provably surrounds the change.
The context bundle from stage 3 is built once for the whole pull request. The reviewer then makes one call per changed file: each call carries that same bundle as reference, plus one file's diff as the code actually under review. Each finding must quote the exact line it is about. LiteLLM is used as a library, so the model is one line of config — Claude, OpenAI, Azure, Bedrock or a local Ollama model — and the key comes from an environment variable.
Why the target is one file, not the whole diff: a single pass over a large diff reports the few most obvious issues and stops looking. The reference material is not what causes that, so the bundle stays whole; only the code under review is kept small, which is why the fifteenth file gets the same attention as the first.
Why the quote: it is the evidence the next stage checks. A model can claim anything, but it cannot copy a line that was never written.
Nothing reaches the pull request without clearing five checks, ordered cheapest first so paid calls are only spent on candidates that could survive.
| Check | How it works | Why |
|---|---|---|
| Quote exists | Whitespace-insensitive text match against the diff and context | Removes invented code before anything else runs |
| Line is in the diff | The reported line must fall inside a changed hunk of that file | A quote can be real while the line reported with it is wrong, and no comment can be placed on a line the diff does not contain |
| Not a duplicate | SHA-1 fingerprint of file, surrounding code and category — never the model's wording | A reworded finding on untouched code keeps the same ID, so re-runs don't repeat comments |
| Skeptic pass | A second LLM call judges whether the finding is real and worth fixing; its confidence is multiplied with the first model's. Skipped for the one kind of finding the parser can prove | Averaging lets a confident wrong answer outvote a doubtful right one. Multiplying makes doubt contagious |
| Rank and cap | Sorted by severity weight × confidence, trimmed to a maximum | Keeps reviews short, so the comments people do see are the ones worth reading |
Every rejected finding records which check dropped it and why, which is what makes review quality measurable later.
A cheap call turns the verified findings into the text of one summary comment. Grouping findings that share a cause — three instances of the same missing null check — reads faster than three bullets that look unrelated.
The guard: this is the only text on the pull request the gate never checked itself, so the summarizer may only group and rephrase findings that already passed — never add a claim. A summary naming a file or line outside the verified set is discarded, and any error falls back to a plain formatted list. A bad call costs wording, never the review.
The call belongs to the core, not to a platform adapter, and its text travels as a summary field in the same JSON. Every adapter gets it without making a model call of its own.
One summary comment is created or updated, plus one comment on the code for each finding.
Why a separate adapter: it depends only on the JSON the command prints, not on the tool's internals. Supporting GitLab or Bitbucket is then glue code for a different API, not a redesign.
One pull request, traced through every stage. calculate_discount gains a required promos argument. checkout.py is not part of the pull request and still calls it with one argument — the break a diff-only review cannot see.
Stages 1–2 · the change that arrives
diff --git a/invoice.py b/invoice.py
@@ -1,6 +1,7 @@
-def calculate_discount(price):
- return price * 0.9
+def calculate_discount(price, promos):
+ rate = 1 - sum(p.rate for p in promos)
+ return price * rate
-def invoice_total(price):
- return calculate_discount(price)
+def invoice_total(price, promos):
+ return calculate_discount(price, promos)Parsed into one file, added lines 1, 2, 3, 6 and 7, and a single hunk spanning lines 1–7. Those spans are what stage 5 later checks a reported line against.
Stage 3 · the context assembled
invoice.py#L1-3 (changed: signature changed)
def calculate_discount(price, promos):
rate = 1 - sum(p.rate for p in promos)
return price * rate
invoice.py#L6-7 (changed: signature changed)
def invoice_total(price, promos):
return calculate_discount(price, promos)
checkout.py#L4-5 (caller of calculate_discount, signature changed)
def checkout(price):
return calculate_discount(price) <-- still the old call
3 blocks, 68 tokens, nothing cutNo model has been called yet. The signature changed between base and head, so every caller of calculate_discount is promoted to must-include — which is how the untouched checkout.py reaches the reviewer at all.
Stage 4 · what the model proposes
{"findings": [
{"file": "invoice.py", "line": 1, "category": "correctness", "severity": "high",
"confidence": 0.9,
"title": "New promos argument is required, but checkout.py still calls calculate_discount(price)",
"quoted_code": "def calculate_discount(price, promos):"},
{"file": "invoice.py", "line": 2, "category": "correctness", "severity": "medium",
"confidence": 0.8,
"title": "Empty promos list is not handled",
"quoted_code": "rate = 1 - sum(p.rate for p in promos or [])"}
]}Note where the first finding sits: on the changed line in invoice.py, naming checkout.py in its text. A finding must live inside the diff, so a break in an untouched file is reported against the change that caused it.
Stage 5 · what the gate does with them
candidate 1 quote found in the diff ....... pass
line 1 inside hunk 1-7 ......... pass
fingerprint 596cc47b1d8b ....... new
skeptic: real, actionable, 0.9 .. 0.9 × 0.9 = 0.81
→ posted
candidate 2 quote found in the diff ....... FAIL
→ dropped at the quote check, no model call spentThe second quote looks plausible and appears nowhere in the diff: the real line has no or []. It is rejected before the skeptic pass, so the fabrication costs nothing.
Stages 6–7 · what lands on the pull request
## Groundtruth review
**1 verified finding**, grouped by file.
#### `invoice.py`
| Severity | Line | Finding | Confidence |
| --- | --- | --- | --- |
| **HIGH** | 1 | New promos argument is required, but | 0.81 |
| | | checkout.py still calls calculate_discount() | |
<sub>1 candidate finding(s) did not survive verification.
Reviewed with `anthropic/claude-sonnet-5` · estimated $0.0032.</sub>The summary call is handed the one verified finding and returns a sentence for the top of the comment — here, “A required argument was added to calculate_discount without updating its caller.” It names only invoice.py, a file in the verified set, so it passes the grounding check; a sentence naming anything else would be discarded and the plain list posted instead. Alongside the comment, one inline comment lands on invoice.py:1 with the severity, category, confidence and quoted line.
Most AI reviewers optimize for coverage — say something about every changed line. Reviewers learn to tune that out fast, and the first time a bot invents a bug that isn't there, every real finding after it gets the same skeptical read. Groundtruth is built around the opposite bet: a quiet, accurate bot earns more trust than a chatty one, because a wrong comment costs more than a missed one.
We read how Qodo, CodeRabbit and Open Code Review describe their own designs and compared them on two questions: who decides what code the model sees, and what stands between a model's claim and your pull request. This is a map of design choices, not a scoreboard. We have not benchmarked any of them against Groundtruth, and a different design is not a worse one.
Placement is our reading of each project's published description, not a measurement
Code decides. Tree-sitter finds the changed function and a text search finds its callers and what the new code calls; a changed signature pulls its callers in. The same diff gives the same context, and the cost is known before any model call.
Rules pick and group the files to review. Then an agent reads whole files, searches the codebase and looks at the other changed files.
Builds a map of definitions and references and keeps an embedding index of functions, tests and past changes, with linters running alongside. The review agent then checks its assumptions with shell commands.
Removed its codebase index in version 2.4. The agent now finds context itself with git diff, grep and file reads, backed by an index of past pull request feedback and team conventions.
The same diff gives the same context, you can see exactly what was sent with --dry-run, and the bill has a ceiling. The price: it only follows paths we wrote. Callers are followed two steps (three at most), found by text search in 18 file types, and callees are found by their definition line, so a bug further out or in another language is out of reach.
An agent can follow a thread nobody wrote a rule for, in any language. The price: what the model saw can differ between two runs of the same diff, and how much it reads is the model's call.
Code checks, then a second model with a veto
The finding must quote code that exists in the diff, on a changed line, and not repeat an earlier comment. A second model then judges it. Its confidence is multiplied with the first model's, and “not real” or “not worth fixing” zeroes it. That second model can come from a different provider, and every drop is recorded with its stage and reason. The exception is a break the parser can prove from the syntax trees, which has nothing left to ask.
Evidence from running checks
Before commenting, it runs shell and Python checks (grep, ast-grep) in an isolated environment to confirm an assumption, so a comment can carry the proof it found.
Separate accuracy modules
Independent modules work on where a comment lands and on what it says. The public README doesn't detail how findings are filtered or deduplicated.
Not described in what we read
The 2.4 write-up says nothing about a verification or false-positive step. One may exist elsewhere in its documentation; we only read the sources listed below.
We follow callers two steps out (three at most) and add the definitions of functions the new lines start calling, but by fixed rules, each capped and dropped first when the budget is short. A bug further out than that, or in a file type outside the 18 we search, is out of reach. Qodo, CodeRabbit and Open Code Review give an agent tools to grep and read as far as it decides it needs to.
CodeRabbit runs linters and security analyzers beside the model, and confirms a suspicion by executing a command and attaching what came back. We run nothing: executing a pull request’s own tooling inside your CI means running whatever the pull request put there. We do one check without a model, by parsing. When a Python signature changes, each caller is tested against the new parameters, and a call that can’t bind is reported with the proof. Every other claim, such as “this call can return None”, is judged by a second model reading the code, not confirmed by running it.
Qodo keeps an index of past review decisions and team conventions and draws on it. On GitHub and GitLab we now record thumbs up, thumbs down and resolved threads on our earlier comments, and the optional server reports them per category, so you can see that one kind of finding keeps getting a thumbs-down. But nothing applies that to the next review: you read the numbers and turn min_confidence up by hand. Doing it automatically is deliberately not built, because a threshold that tunes itself on thin, noisy data can quietly make the reviewer worse.
Read in October 2026 from each project's own pages: Qodo, on removing most of its RAG system in 2.4; CodeRabbit, on accurate reviews on massive codebases; Open Code Review README.
1 Several GitHub projects share this name; this is Alibaba's. Products change, so if something here is out of date, please open an issue.
Six pieces, each independently true of the code today — not a roadmap.
LiteLLM is embedded as a library. Claude, OpenAI, Azure, Bedrock, NVIDIA's API catalog, any OpenAI-compatible endpoint, or a fully local Ollama model. The model is one config line; the key and the endpoint come from your environment, never from a file the pull request can edit.
A changed line expands to its whole function with Tree-sitter; its real callers, and what the new code calls, are found by text search. A changed signature promotes those callers to must-include, even in files the pull request never touched.
Quote match, line-in-diff, dedupe fingerprint, an adversarial skeptic pass with multiplied confidence, then rank and cap. Every rejection is recorded with the stage and the reason.
Review with Claude and verify with GPT, or the other way round. A model is a poor judge of mistakes it would make itself, so the skeptic pass can run on a different provider entirely — one config line, verify_model.
GitHub, GitLab and Bitbucket Cloud adapters share one core and one JSON contract. The published image runs the same command as a step in Tekton, Argo Workflows or Jenkins on Kubernetes.
Labeled cases replay offline against recorded model responses. Every expected finding lands as caught, gated or missed — two very different failures that look identical from the outside.
The whole install: one repository secret, one short workflow file. No server, no database, no webhook to register.
# .github/workflows/groundtruth.yml
on: pull_request
# A second push cancels the review still running for this PR, instead of
# racing it to write the summary comment last.
concurrency:
group: groundtruth-${{ github.event.pull_request.number }}
cancel-in-progress: true
permissions:
pull-requests: write
jobs:
review:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with: { fetch-depth: 0 }
- uses: groundtruth-code-review/groundtruth-code-review/adapters/github@main
with: { model: anthropic/claude-sonnet-5 }
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}Add ANTHROPIC_API_KEY as a repository secret and that's the complete install. GitLab, Bitbucket Cloud and the container image are documented in the README, and --dry-run shows the assembled context with no key and no model call at all.
Three stages call a model, and they do not need the same one. The model is a config line, so the tier is a deployment choice rather than a code change.
| Stage | Tier | Why |
|---|---|---|
| 4. Propose | Stronger | Reads a whole file's diff against the context bundle. The ceiling on review quality is set here |
| 5. Verify | Cheap | One bounded question about one finding and its evidence, on a short prompt |
| 6. Summarize | Cheap | Rephrasing text that is already verified, once per pull request |
Why the reasoning gets the stronger tier: a finding that was never proposed cannot be recovered downstream, while a finding that was proposed still has to get past the gate. Spend where the reasoning happens, not where the checking does.
Why the verifier can be a different provider: the skeptic pass is only as independent as the model running it. Two calls to one model share the same training and the same blind spots, and a model asked to judge its own kind of output tends to agree with it. Propose with Claude and verify with GPT — or the reverse — and the second opinion comes from a model that learned differently, so it is less likely to share the first one's specific mistake. The verifier is never told which model proposed the finding.
The cheap tier is safe in stage 5 because most of that stage is not a model call at all. The quote match, the location check, the fingerprint and the ranking are ordinary code — exact, free, and unaffected by which model is configured. Only the skeptic's judgement is a call, and its confidence multiplies, so an unsure verdict pulls a finding below the bar rather than waving it through.
Several obvious pieces are missing on purpose. Each was left out for a reason worth knowing before adding it back.
| Not used | Used instead | Reason |
|---|---|---|
| Vector database, embeddings, RAG | AST parsing plus a caller search | Retrieval guesses at relevance; a wrong snippet turns into a confident, wrong finding |
| A hosted service or shared API key | The user's own key from an environment variable | No shared bill, no contract, and a local Ollama model costs nothing |
| Server, queue and database for normal use | A stateless CI job | One person reviewing their own repository needs a key, not a fleet |
| Spend governance built into the tool | An optional self-hosted LiteLLM proxy, set by one environment variable | Budgets and caps already exist in that open-source proxy; rebuilding them would add no value |
| Keys in a config file | Environment variables only, with the config file refusing to load anything key-shaped | A file that could hold a secret is a file someone eventually commits |
| Averaged confidence scores | Multiplied scores | A false positive costs more trust than a missed bug, so either model's doubt must be able to block a finding |
One more boundary holds the design together: the context and verification packages never import platform code. They take a diff and a repository path, and return findings. That is what makes “any platform, any model” true rather than a slogan.