Browse the book & resources

Supporting material

Desk-researched interview 3: Who corrects AI-assisted changes?

Desk research, not original interviews. These question sets synthesize published accounts; no practitioner was interviewed for this book.

Question: What percentage of AI-assisted changes required substantial correction, and who performed that correction?

Method: Desk research from an empirical open-source PR study, published employee interviews and engineering accounts; no team-specific correction audit was conducted. Research snapshot: November 2026.

What can actually be counted

Watanabe and colleagues studied 567 Claude Code–identified PRs from 157 open-source projects, paired with 567 human PRs. 475 of 567 agentic PRs merged; of those 475, 261 (54.9%) had only the initial commit, while 214 (45.1%) had at least one later commit. The latter is a revision rate among merged PRs, not the percentage of all AI-assisted changes needing substantial correction. Revisions can be minor, and the study cannot see corrections made before the initial PR. The authors found no statistically significant difference in measured revision effort between these agentic and matched human PRs. Watanabe et al., On the Use of Agentic Coding, 2026, §§3–4.3.

The same paper classifies purposes of revisions, including bug fixes, documentation and refactoring, but the recorded PR history does not reliably attribute each correction to the original author, another reviewer, or a coding agent. Anthropic's internal interviews describe engineers actively validating and cleaning up Claude output; one respondent described shifting toward code review/revision. These are individual accounts, not a denominator for correction frequency or an organization-wide labor allocation. Anthropic, How AI is transforming work at Anthropic, 2025.

Synthesis, not testimony

Answer: The published sources cannot supply the requested substantial-correction percentage or who did all of it. The defensible nearby figure is 45.1% of merged agentic PRs had a subsequent commit in one selective open-source sample. Do not relabel that figure as a substantial-correction rate, an AI failure rate, a revert rate, or a measure of all enterprise AI-assisted work. The primary outcomes at risk are reliable quality and efficient verification effort; reviewer burden also affects engaging.

For a local audit, define a substantial correction before measuring: e.g., a change to behavior, security boundary or architecture required to make a PR acceptable, excluding formatting or requested scope additions. Record the denominator (all submitted AI-assisted changes, including rejected ones), provenance, change risk, reviewer-requested rework, post-merge fixes and time spent. Have independent reviewers classify a sample and reconcile disagreements. Identify who found, specified and implemented each fix; these may be different people.

Questions to ask a practitioner

  1. Of the last 100 AI-assisted changes, how many met your pre-agreed substantial-correction definition, including rejected submissions?
  2. For each correction, who detected it, who wrote the fix, and how many reviewer and author hours were needed?
  3. What is the comparable rate for similarly sized, similarly risky non-AI changes?

Follow-up desk-researched responses

1. How many of the last 100 met a substantial-correction rule? Unknown for any specific team's last 100. Watanabe et al.'s nearest published count is 214 of 475 merged agentic PRs (45.1%) with a second or later commit. That is neither a sampled last-100 audit nor a coding of substantial corrections; it omits rejected PRs and pre-submission edits. Salesforce's different BugWiser report says around 90% of bug-classification predictions were accepted unchanged; that measures model classifications, not AI-assisted code changes. PR study, §§3–4.3; Salesforce BugWiser account.

2. Who detected, specified and fixed them, and how many hours? The PR study describes subsequent commits and categories of revisions, not a reliable allocation of detection, specification, implementation and review hours by role. Salesforce's review-system account says human reviewers retain final scrutiny with automated assistance, while BugWiser allows engineers to correct classifications; neither is a time-and-motion ledger for code fixes. PR study; Salesforce review-system account; BugWiser account.

3. Compared with similar non-AI changes? In the study, 214 of 516 merged matched human PRs (41.5%) also had a later commit (516 merged minus 302 single-commit PRs). The authors report no statistically significant difference between groups in revision frequency or measured revision effort. Matching by author/repository does not fully equalize task risk or size; neither figure is a substantial-correction rate. Watanabe et al., §§3–4.3.

Evidence gap: No cited study measures this team's substantial-correction percentage or distributes its correction hours by role. The open-source sample includes only PRs explicitly marked as Claude Code–generated.