AI Code Checkers: What They Catch, and What Acceptance Still Needs

What is an AI code checker

The term covers five kinds of tool. Rule-based analyzers existed long before language models and now use one to explain a finding or propose a fix. Language-model reviewers read the diff, pull in context from the repository and comment the way a colleague would. Dependency checkers look at the packages a change brings in. Test-generating checkers write tests for the change and run them. Conformance checkers hold the change against a written specification or architecture rule. The useful comparison is by what each one reads, because a checker can report only on what is in front of it.

The last column is about evidence: a finding nobody can reproduce carries little weight later, so record the model, version and configuration behind each language-model review.

What the studies say AI code review catches

Two peer-reviewed studies measure AI code review in real use, one industrial and one across open-source repositories, and a third, from before the bots, measures what human review is for. They count different things, which is worth knowing before quoting any of them.

An industrial rollout. Cihan and colleagues followed a language-model reviewer that commented on every pull request in three projects at a multinational appliance maker, 1,568 pull requests reviewed out of 4,335 (ICSE-SEIP 2025). The headline result: "73.8% of automated code review comments were labeled as resolved. However, the overall average pull request closure duration increased from five hours 52 minutes to eight hours 20 minutes, with varying trends observed across different projects." The authors write that "while developers invested additional effort to address automated comments, it did not replace the need for human reviews", and they "did not find conclusive evidence supporting consistent time or effort savings". One practitioner named the risk at the center of this piece: "It may create bias so reviewers may ignore by saying that if any other issue exists, the bot would have written it." A caveat: "resolved" was a label developers set under a mandatory policy, and the authors note some may have used it loosely.

Open-source repositories. Sun and colleagues studied the 16 most popular AI review actions for GitHub workflows, across 178 repositories and more than 22,000 AI comments (IEEE Transactions on Software Engineering, 2026). Language models, checked against human labels, judged whether each comment was valid and followed by a code change before merge. Across the four actions with enough data, 0.9% to 19.2% of AI comments met that bar, against 60% of human comments in the same repositories. For a team choosing a tool, the spread is the useful part: actions that comment on specific blocks of changed lines were acted on more often (6.5% to 19.2%) than actions that comment on whole files (0.9% to 4.2%). The actions were selected in January 2025, and the projects are mostly small to medium.

What review was for before the bots. Bacchelli and Bird classified 570 review comments and surveyed 873 programmers (ICSE 2013). Finding defects was the first motivation for 44% of the programmers. Defects were the fourth most frequent of nine kinds of comment, at 14%, behind code improvements at 29% and understanding in second place. Review, they wrote in their abstract, provides "additional benefits such as knowledge transfer, increased team awareness, and creation of alternative solutions to problems."

Read together: checkers produce findings worth fixing, teams act on a share that depends on the tool and on how specific the comment is, and closing a change can take longer. Each study counts comments, and none was designed to say whether the merged change was the right one.

What acceptance needs on top of the checkers

Our white paper The Agentic Engineering Assurance Layer puts the place of every checker in one passage. It counts review bots, static analyzers, security scanners and model evaluators among the sources of findings, and then:

They can be highly specialized and may provide stronger evidence than a general coding agent. Their result remains a contribution to a decision. A pass can be irrelevant if the wrong subject, environment, policy version, or acceptance criterion was tested. A finding source should not silently become the final authority for the work it evaluates.

The decision those findings feed has four inputs that no checker produces by itself. Our article on agentic engineering explains why that decision now sets the pace of agent-built work.

What the change was for. A checker compares code with rules, with the rest of the codebase or with tests. Whether those were the right things to compare against is a question of validation, and AI Has Moved the Bottleneck states the limit plainly: "A green pipeline can establish that the implementation matches the test surface. It cannot establish that the test surface represents the whole intent." The intent has to be written down before the change, where a reviewer can read it, by someone other than the tool that made the change.

Who checked it, and how far from the producer. The same paper starts from the producer: "The entity that produces the work should not be the sole entity attesting that the work is correct." How far apart the two should stand depends on what failure would cost, and the paper is direct about that too: "The correct degree of independence is a risk decision. The relationship must be at least visible." At the far end of that range sits independent verification and validation, as NASA's NPR 7150.2D states it, citing ISO/IEC/IEEE 24765: "Verification and validation performed by an organization that is technically, managerially, and financially independent of the development organization." Most changes need far less, and every one of them needs the relationship written down: who or what produced it, which checker read it, and what the two had in common.

Who has the authority to accept. The standards vocabulary treats acceptance as a decision made by somebody entitled to make it. Acceptance criteria are "criteria that a system or component is required to satisfy to be accepted by a user, customer, or other authorized entity", in the wording of ISO/IEC 33202:2024 as published in SEVOCAB. A checker supplies evidence against those criteria, and the entity weighs it. AI Has Moved the Bottleneck adds a condition: "Human approval is not meaningful if the reviewer has no time, evidence, or authority to challenge the result." A reviewer facing a long list of bot comments and a deadline holds the authority on paper only, which is the bias the practitioner in the industrial study described.

A record that outlives the session. The findings, what was done about each, and the decision must be readable later by somebody who was not there. In the paper's words: "A later reviewer should not require access to the original model conversation or the memory of the engineer who was present." Review threads can be edited, branches deleted and bot comments collapsed. A team that may have to explain a change to an auditor keeps the record somewhere tamper-evident.

A code review checklist for agent-written changes

NIST's Secure Software Development Framework already keeps the two activities apart. Practice PW.7.1 asks an organization to determine whether to use "code review (a person looks directly at the code to find issues)" and/or "code analysis (tools are used to find issues in code, either in a fully automated way or in conjunction with a person)", and PW.7.2 asks it to "record and triage all discovered issues and recommended remediations in the development team's workflow or issue tracking system" (SP 800-218). Its profile for generative AI declines to make machine-written code a special case: "Practices and tasks in this Profile do not distinguish between human-written and AI-generated source code, because it is assumed that all source code should be evaluated for vulnerabilities and other issues before use." (SP 800-218A)

The checklist below follows the same principle and fits any change. It is written for the reviewer of a change that an agent produced or helped produce, from any of the four categories of AI coder, where some of these questions are easy to skip. Each question names where its answer should already be. If an answer has to be pieced together from memory or a chat history, treat the question as open.

Two questions need a word more. Authorship matters because tests written by the agent that wrote the code can share its reading of the task, and that reading is what a test exists to check. Dependencies matter because language models sometimes name packages that do not exist, and anyone can register such a name and publish code under it. Spracklen and colleagues measured how often, on 16 models tested in 2024: of 2.23 million recommended packages, "440,445 (19.7%) were determined to be hallucinations", and when the same prompts were asked again, "43% of hallucinated packages were repeated in all 10 queries" (USENIX Security 2025). A name that comes back every time is a name worth registering. Newer models were not part of that test.

The checklist applied to one public pull request

F Prime is an open-source flight software framework, originally developed at NASA's Jet Propulsion Laboratory and deployed on several space applications. Its rules already cover several items on the checklist. The AI policy says contributors "must disclose all generative AI usage", the contributing guide says "an approved review from a maintainer will be required to complete the submission", and the repository has its own multi-agent AI review, which ran twice on this pull request.

Pull request 6032, merged on 28 September 2026, fixes a fault in how its file-transfer component receives files sent from the ground: two malformed messages could trap the component's thread in an endless loop. An autonomous coding agent produced it, a person reviewed it before submission, and the agent's statement says it "wrote the fix and the two regression tests, validated them against patched and unpatched builds, and iterated on the automated review feedback". The public record answers the checklist like this.

By open-source standards this is a careful change: a precise problem statement, regression tests, automated reviewers that caught a real documentation gap and a design inconsistency, and two maintainers signing off. An acceptance decision in software whose failure has consequences would still ask four things the record leaves open. Which reviewed need does the change answer? Does a test written apart from the producer show the failure? Who settled the open disagreement, and on what grounds? What did each approval rest on? Each would have taken a sentence to record while the work was fresh.

Where Kealu Vector fits

The Agentic Engineering Assurance Layer is the proposed category for the layer that governs how agentic engineering work is performed and evidenced, so that humans can trust and accept the resulting work. The white paper The Agentic Engineering Assurance Layer develops it as a vendor-neutral category, and the questions raised here hold whatever tools a team uses to answer them. Kealu Vector, currently designated Vector Assurance Assist, is our early-access implementation of that category.

Frequently asked questions

What is an AI code checker?

An AI code checker is a tool that reads a proposed change to a codebase and reports problems in it, using a language model on its own or together with rule-based analysis. The five common kinds are rule-based analyzers with model-written explanations, language-model reviewers, dependency checkers, test-generating checkers and conformance checkers.

Can AI code review replace human code review?

AI code review adds findings, and teams act on a share of them that depends on the tool and on how specific each comment is. Accepting the change still belongs to a person with the authority to do it, who can explain the decision later. Human review also spreads understanding of the code among the people who will maintain it.

What should a code review checklist for AI-generated code include?

Ten questions, from the need behind the change to where the record is kept. The table above gives each one, with where its answer should already be.

Is an AI code checker the same as verification?

Verification, in the standard vocabulary, is "confirmation, through the provision of objective evidence, that specified requirements have been fulfilled" (ISO/IEC/IEEE 12207 and 15288, as published in SEVOCAB). A checker's findings are part of that evidence, for the rules or tests it applied. Validation, whether the change serves its intended use, needs a person who knows that use.