nodrail

All guides

How we blocked a GitHub merge after an AI agent reviewed an older commit

By Nodrail · 4 min read · Published

On this page
  1. What QA was asked to review
  2. Four decisions on one pull request
  3. Where the policy ends and GitHub takes over
  4. What this means for a team

QA reviewed a quota change and reported a pass. We pushed one more commit to the same pull request. It changed only the formatting of a JSON file. Every other GitHub check passed, and the configured quota still behaved exactly as QA had described. Should the old review count?

Our policy required QA to review the commit being merged. On November 15, 2026, we tested that requirement on a temporary protected branch in Nodrail’s public agent plugins repository. GitHub held the second commit until QA reviewed it. This GitHub merge check was an operator-run pilot. You can inspect the two commits and the closed pull request. Nodrail/agent-plugins PR #11

What QA was asked to review

The change added one file, examples/apf-pilot/limit.json. Its quota was three calls, counted from one, with the limit included. Call three should go through. Call four should be held.

Our runner fetched the file from GitHub by commit SHA and gave its values to Claude Code, using our existing QA instructions. The agent reported a pass and explained the boundary. Separately, the runner checked calls one, three and four against the committed configuration. Those were two pieces of evidence. The receipt recorded the agent’s judgment. The boundary check tested a concrete claim we could verify without taking the agent’s word for it.

The target branch required the repository’s existing Check plugins job and a second status named APF / revision receipt. We kept the first job in place so we could see whether the QA requirement made a difference after ordinary CI passed.

Four decisions on one pull request

Current PR headQA evidence submittedGitHub merge state
05f39aeNo receipt for this commitBLOCKED
05f39aeQA passed on 05f39aeCLEAN
3fed234The same passing receipt from 05f39aeBLOCKED
3fed234QA passed on 3fed234CLEAN

The third row is the one to inspect. The second commit spread the same JSON values across four lines. After the push, GitHub initially waited for a status on the new head. We then deliberately supplied the first receipt. The adapter rejected it because its recorded head was 05f39ae. Even after every other check turned green, GitHub still reported BLOCKED. Formatting-only commit

We ran QA on 3fed234, repeated the boundary check and posted the passing status to that SHA. GitHub reported CLEAN, meaning the PR was eligible to merge. We closed it without merging and removed the temporary branches.

Where the policy ends and GitHub takes over

A gate belongs in the same Agent Policy Format (APF) YAML document as the team’s permission rules. Its job here was to require a passing QA step before the covered merge action could proceed. Receipts provide run evidence, and the pilot adapter decided which evidence applied to this request. Agent Policy Format · Runs, gates and budgets (preview)

For this pilot the adapter compared four values with the receipt: repository, pull-request number, base SHA and head SHA. Only a matching receipt with a passing independent check supplied QA evidence to the gate. The adapter posted the resulting status to the exact head it evaluated. GitHub enforced the merge restriction because the target branch required that status. An agent-side hook alone cannot hold a merge initiated in the GitHub web interface. GitHub’s protected branch documentation

What this means for a team

An old receipt remains useful. It tells a reviewer which revision QA examined and why QA passed it. When a new commit arrives, the actionable message is simple: “QA passed on 05f39ae. The current commit is 3fed234. Run QA on this commit.”

Requiring an exact match has a cost. Our formatting change triggered another review even though the behavior stayed the same. A team choosing this policy is deciding that every new head warrants fresh evidence. Our pilot exercised that choice on one PR, using locally triggered checks. An installable GitHub App, receipts from an authenticated issuer, and behavior in merge queues or across PRs sharing a commit are still engineering work.

If your decision happens earlier, when the agent tries to open a PR or push code, the enforcement point is the covered tool call. If your decision is whether GitHub should merge an already open PR, check what the target branch requires for its current head. QA gates for opening a pull request (preview) · Agent receipts and run history (preview)