What is the Harness-IF study and why does it matter for AI code review?

Harness-IF is a study by ByteDance Seed and Tsinghua covering 60 multi-turn coding items drawn from a library of 642 rules, run across twelve frontier models on five configurable surfaces. Raw compliance accuracy ranges 72.1% to 85.9%, but once you isolate the rules a model would not have followed on its own, Against-Prior Accuracy drops to 66.1% to 78.6% — a 3.6 to 7.4 point fall, averaging 5.81 points. It shows that an aggregate compliance number overstates how well a reviewer enforces rules that actually differ from the model's default behavior.

How do you test whether an AI reviewer actually loaded your rules file?

Use a canary test: put a made-up marker in your rules file, like a line requiring every review to begin with a specific string, then open real pull requests and check for it. Run the test twice on separate sessions with a different token each time, since a reviewer that fetches once and caches the file will pass a single canary and then run stale indefinitely. Passing the canary only proves the file arrived, not that the model applies it.

What is the against-prior test and how do you run it?

Build two small rule sets: one of against-prior rules, which a model would not follow unprompted (like requiring all DB access through a repository layer), and one of free-compliance rules the model would satisfy anyway (like descriptive function names). Run both sets and compare the scores. A tool that scores 90% overall but 55% on the against-prior set has a reviewer with a good demo and a weak gate on your actual standards.

Does rule compliance hold up over a long coding session?

No. MTAC-IFBench, from Tsinghua's CoAI group, Zhipu AI, and UESTC, tested 11 code agents across 100 multi-turn instances with 7.04 turns each and found that strict per-turn constraint compliance (C-ISR) is mostly below 10% on average and collapses toward zero by turn six for most models, even though functional success stays high. GLM-5.2, the best model tested, dropped from 12.7% average C-ISR to 2.6% on turns nine and ten.

Does it matter whether rules live in a config file versus a chat instruction?

Yes. MTAC-IFBench found that constraints held in repository policy files held up better over a long session than the same constraints delivered as user instructions, and that updating an existing constraint mid-session was harder for agents to track than adding a new one. A rule file on disk gets re-read and re-anchored; a correction given at turn nine has to be held in working memory while the agent does everything else.

AI code review that follows your coding rules: how to test it

A reviewer that loads your rules file is not a reviewer that enforces it. Two instruction-following studies give you three tests that tell the two apart.

The number that should make you suspicious of any compliance claim

An agent that obeys a rule may simply have been going to do that anyway.

That sentence is the premise of a ByteDance Seed and Tsinghua study called Harness-IF, and it is the most useful thing I have read this quarter for anyone evaluating an AI code reviewer against their own standards. The authors took 60 realistic multi-turn coding items drawn from a library of 642 rules, pushed 256 of those rules out for verdicts, and placed them on the five configurable surfaces a deployed agent actually reads. Twelve frontier models went through it.

Raw accuracy across those models spans 72.1% to 85.9%. Looks fine. Then they scored only the rules labeled as opposing the model’s unprompted default, re-running each task with the rule withheld across nine probe builds to confirm the label. Against-Prior Accuracy lands at 66.1% to 78.6%. Every model got worse on those rules, by 3.6 to 7.4 points, mean 5.81. The top build kept its spot, but three adjacent rank pairs swapped.

So the aggregate compliance number a vendor quotes you contains a pile of rules the model was already going to follow. The slice that matters, where your team’s convention differs from the model’s default, runs about 5.8 points lower on average, and the size of that gap is model-specific, which means you cannot apply one correction factor across tools. You have to measure the gap on your own rules.

This is the second version of this page. The first covered the config-loading question and the canary test. What I did not have then was published data on how compliance behaves once a session gets long, and that turns out to be where teams lose the most.

Canary first: did the rule reach the model at all?

Before you score obedience, prove receipt. Put a made-up marker in your rules file, something like Every review must begin with the line: REQUIRES-DIVISOR-K7, and open a few pull requests with real diffs. A model that never loaded the file cannot produce that string. Run it twice on separate sessions with a different token the second time, because a loader that fetches on session one and caches will pass a single canary and then run stale forever.

I have the full protocol for that, including block tests with plausible violations, in how to verify your AI reviewer follows your team’s rules. The short version: passing the canary proves the file arrived. It says nothing about whether the reviewer applies it.

The prior-control test: score only the rules the model would have broken

Harness-IF’s second contribution is a test design you can copy without any of their tooling. Pick rules, then ask for each one whether a competent model would follow it unprompted. A rule like “functions should have descriptive names” is free compliance, and scoring it tells you about the model’s training, not your standards. A rule like “all DB access in this module goes through the repository layer” is against-prior: nothing in the model’s default behavior produces it, and a reviewer that ignores your config will let it slide.

Build two small test sets, one of against-prior rules and one of rules the model would satisfy anyway, and run both. The gap between the two scores is the honest measure of your rules config. If a tool scores 90% overall and 55% on your against-prior set, you have a reviewer with a nice demo and a weak gate.

The study’s conflict pilot adds one more thing to test: precedence. Across nine separate builds, pooled precedence did not follow prompt depth. System prompts, project files, and user instructions all sat ahead of tool and skill descriptions. If your reviewer injects your rules through a surface that loses to something else in the stack, the rule can be present, correct, and outvoted.

The turn axis: a rules file read at turn one decays by turn six

A second paper, MTAC-IFBench from Tsinghua’s CoAI group, Zhipu AI, and UESTC, gives the missing dimension. It scores 11 code agents across 100 multi-turn development instances with 7.04 turns each and 91.33 constraint items per instance, 12.97 per turn, 9,133 constraints in total. Every constraint has a checklist item, verified either by a generated script or by a judge agent, with Claude Code as the standard harness.

They report two metrics that should both appear in any review-tool evaluation. Constraint Success Rate is the share of constraints satisfied in a turn. Constraint-Instruction Success Rate requires every constraint in that turn to be satisfied. Average CSR across models runs 60.6% to 80.4%. Average C-ISR is mostly below 10%.

Concrete rows from their table, under Claude Code:

GLM-5.2, the best model on the board: CSR 80.4%, C-ISR 12.7%. On turns 9 and 10, C-ISR drops to 2.6%.
Claude-Opus-4.6: CSR 78.6%, C-ISR 8.7%. Turns 9 and 10: 2.3%.
Gemini-3.1-Pro: C-ISR 11.1% overall. Turns 5 through 10: 0.0%.
Qwen3.6-Plus: C-ISR 2.0% overall, 0.0% from turn 3 onward.
Claude-Haiku-4.5: C-ISR 1.9% overall.

Most models hit zero strict compliance by turn six. The functional side looks completely different: Claude-Opus-4.6 built a working project 98% of the time and satisfied 90.2% of function checklist items, while its strict function rate was 37%. Gemini-3.1-Pro satisfied 66.5% of functions and 14% strictly.

Read that alongside the harness study and the practical implication is blunt. A reviewer that checks the final diff is checking the artifact at the point where process compliance has already fallen off a cliff. The violations your team cares about, the naming rule, the required error path, the migration guard, mostly happen during the work, and an evaluation that only looks at the merged state cannot see them.

Where the rule lives changes what the agent does with it

MTAC-IFBench’s fourth finding is the one I keep coming back to for config design: constraints held in repository policy files held up better over a long horizon than the same constraints delivered in user instructions. And inside user instructions, updating an existing constraint was harder than adding a new one. Agents track new requirements better than they track changed ones.

If you have ever watched an agent follow a rule for forty minutes and then quietly revert to its own convention, that is the mechanism. A rule file on disk is re-read and re-anchored. A correction buried at turn nine is a request the model has to keep in working memory while it does everything else.

The paper’s own framing of where agents are weakest matches what I see in practice: isolated static constraints like file encoding get followed, while project-wide consistency, multi-step coordination, and precise quantitative limits get missed. Those are exactly the constraints a reviewer is supposed to enforce.

What this means for picking a reviewer

The tools differ less in whether they accept rules than in how deep the rules surface goes and whether the reviewer re-reads it every run. From the tool directory, the range is visible. Tabnine’s Code Review Agent checks PRs against plain-language team rules on GitHub, GitLab, and Bitbucket. Bito routes custom guidelines through its own config. CodeRabbit scopes instructions to paths and lets team-wide rules become named pre-merge checks. Kodus takes org-wide rules defined once and applied across the repos it reviews, self-hosted or cloud with your own model keys. Those are meaningfully different depths of the same feature, and the directory scores each on documented coverage rather than measured behavior, which is why the tests below still decide it.

A protocol you can run in an afternoon

Canary with a made-up marker, twice, on separate sessions. Proves the file reaches the model and is not cached.
Two rule sets, one against-prior and one free-compliance. The gap between the scores is your real config score.
The same violation at turn one and at turn nine of a session. If it holds early and slips late, your reviewer is reading a rules file the agent has stopped honoring.
Scope check: feed the same violation through a directory your rule does not cover. Path-scoped rules should stay quiet.

What rule configuration still cannot cover

A reviewer that follows every rule you wrote is still bounded by the rules you wrote, and there is a structural blind spot underneath the config surface. If the model that generated a patch is the only thing reviewing it, you have a correlated judge, and a single model grading its own output is one opinion measured many times. Rule checks are the layer you want flat and reproducible. The judgment about whether the change is right belongs to something that can disagree with the generator.

The short version

Rules support is a claim about behavior, and behavior has two parts: whether the rule arrived, and whether it survived the session. Harness-IF shows that aggregate compliance overstates the against-prior slice by about 5.8 points on average across 12 models. MTAC-IFBench shows that strict per-turn compliance collapses to near zero within six turns even for the best agent on the board, while functional success stays high. Test receipt, test the against-prior slice, and test late in a session. The tool that holds on your rules at turn nine is worth more than the one with the better demo at turn one.