How to Review AI-Generated Documentation: Catching the Page That's Almost Right

A human-written doc that's wrong usually looks a little wrong. It's thin in the places the author didn't understand, it trails off, it says "TODO: confirm this." AI-generated documentation has no such tells. A model can produce a page about your payment retry logic in thirty seconds that is well structured, consistently formatted, confidently worded — and describes a backoff strategy your code stopped using in March. Reviewing AI-generated documentation is a different job from reviewing a teammate's draft, because the fluency of the prose tells you nothing about whether it's true.

The research backs this up. In a study of LLM-generated code comments, Kang, Milliken, and Yoo found that even the best-performing model produced comments where roughly a fifth contained demonstrably inaccurate statements — and that existing code-comment consistency checks showed no statistically significant relationship with whether a comment was actually accurate. Developers already feel it: in the 2025 Stack Overflow Developer Survey, 46% said they distrust the accuracy of AI tools, and the most common frustration, cited by 66%, was "AI solutions that are almost right, but not quite." Almost right is the defining failure of generated docs.

None of that means you should stop generating documentation. It means review has to be aimed at the right targets. This guide covers how AI-generated docs fail, a review pass that catches those failures in a sensible order, the checks you can hand to a script, and the trap that gets through even a careful review: a page that was correct on the day you approved it.

How AI-generated docs fail#

The failure modes of generated documentation cluster in predictable places, and they're different from the failure modes of human-written docs. Knowing the difference tells you where to spend attention.

The pattern behind the table is that a model writing documentation is predicting what documentation for this code would probably say. When the code does the probable thing, the docs are right. When your code does something unusual — a non-standard default, a deliberate exception, a workaround for a vendor bug — the model drifts toward the probable answer and away from yours. The unusual parts are exactly the parts readers most need documented, which is why generated docs can be 90% correct and still fail at the job.

The same dynamic shows up when the author is a coding agent rather than a doc generator. Our guide to reviewing AI-generated pull requests makes the point that an agent PR's description often restates the prompt instead of summarizing the diff. Generated docs do the equivalent: they summarize what the code is named rather than what it does.

Where review effort belongs#

Not every sentence in a generated page deserves the same scrutiny. Most of it is connective prose that's harmless if slightly off. The claims that matter are the ones a reader will act on — and many of those can be checked by a machine before a person reads anything.

flowchart TD
  A["Docs generated from code"] --> B["Mechanical checks: paths, identifiers, commands, links"]
  B -->|failures| A
  B --> C["Source check: does each cited file say this?"]
  C -->|mismatch| A
  C --> D["Human review: accuracy of behavior, caveats, judgment"]
  D -->|changes requested| A
  D --> E["Merged into the docs"]
  E --> F["Re-checked when the cited code changes"]
  F -->|drift found| A

The first two boxes are cheap and should never consume a reviewer's attention. The third box is where a human earns their keep: deciding whether the page describes the system's real behavior, whether it leaves out a caveat that will hurt someone, and whether it's the right page to have at all. The last box is the one most teams skip, and we'll come back to it.

A review pass for AI-generated docs#

When a generated page reaches you, work through it in this order. The sequence front-loads the questions that let you reject quickly.

1. Check that it's grounded at all. Does the page say which source files it describes? A generated doc without source references is a doc you have to verify from scratch, because you don't know what the model was looking at. If your generator can attach citations, require them. If it can't, the first step of review is reconstructing them, which is slow, and a good reason to pick a different generator.

2. Verify every identifier and path mechanically. Invented names are the cheapest defect to catch and the most embarrassing to ship. Don't eyeball them. Run something like this against the page:

f=docs/payments.md
grep -oE '`[A-Za-z0-9_./-]+\.(ts|tsx|js|py|go|md|json|ya?ml)`' "$f" | tr -d '`' | sort -u \
  | while read -r p; do [ -e "$p" ] || echo "missing path: $p"; done
grep -oE 'npm run [a-z0-9:_-]+' "$f" | awk '{print $3}' | sort -u \
  | while read -r s; do jq -e --arg s "$s" '.scripts[$s]' package.json >/dev/null || echo "no script: $s"; done

Adapt the patterns to your stack — make targets, CLI flags, environment variable names. Anything that comes back from this is a hard error, not a judgment call.

3. Verify every number. Timeouts, limits, retry counts, sizes, defaults, version numbers. These are where "plausible default" errors live, because a model that can't find the value fills in a typical one. Each number in the doc should trace to a line in the source. If you can't find it in under a minute, treat it as wrong.

4. Read the implementation behind each behavioral claim. "Retries failed webhooks up to five times with exponential backoff" is three claims: that it retries, the count, and the strategy. Open the cited file and check each one. This is the step that catches intent-over-behavior errors, where the model described what a function named retryWithBackoff ought to do.

5. Look for what's missing. Generated docs over-represent the happy path. For each operation the page describes, ask what happens when it fails, what it costs, what it requires, and who is allowed to do it. If the code has a guard, a rate limit, or a permission check that the doc doesn't mention, the doc is incomplete in the way that matters most. Runbooks are especially vulnerable here, which is why our runbook guide puts failure modes before procedures.

6. Check diagrams separately. A generated architecture diagram is a set of claims about which components talk to each other, presented in a format reviewers tend to trust more than prose. Check each edge. Paste it into a Mermaid editor if you need to see it rendered, but verify the arrows against the code, not against your memory of the architecture.

7. Only now: read it as a reader. Is it clear, is it pitched at the right audience, is it in the right place? This is the step human reviewers naturally start with, and it's the least important for generated docs, because the prose is rarely the problem.

The diagnostic below maps the claim types you'll find to the cheapest way to verify each one.

Kang, Milliken, and Yoo's paper suggests a more ambitious version of steps 3 and 4 that they call document testing: generate tests from the documentation's claims, run them against the code, and treat failures as evidence the docs are wrong. It's more machinery than most teams need for a single page, but the principle is the one to keep. A documentation claim is only verified when something checked it against the code, not when someone read it and nodded.

Watch: AI docs tooling that cites its sources#

Alex Garnett's talk at Write the Docs Portland 2025 opens with a line most documentarians will recognize — "We do not want AI to write the docs" — and then makes the case that docs AI tooling is genuinely useful when it cites its sources and sends readers back to the material it drew on. The same property is what makes AI-generated docs reviewable: a claim with a source attached can be checked in a minute, and a claim without one can't.

The trap: docs that were true when you approved them#

Everything above catches errors that exist at review time. The defect that gets through is the one that doesn't exist yet. You review a generated page carefully, every claim checks out, you merge it. Three weeks later someone changes the retry count from five to three. The doc is now wrong, nothing flagged it, and it reads exactly as authoritative as the day you approved it.

This is documentation drift, and AI generation makes it worse in two ways. First, volume: a team that generates docs for every module now has far more pages to keep in sync than a team that wrote them by hand. Second, provenance: a human author remembers what they wrote and sometimes notices when the code changes under it. A generator doesn't remember anything. If nothing re-checks the page when its source changes, nobody will.

There's a reason this matters more now than it used to. DORA's 2024 research found that a 25% increase in AI adoption was associated with a 7.5% increase in documentation quality — but the same research found an estimated 7.2% reduction in delivery stability. Teams are producing more docs and better-looking docs, and shipping changes that invalidate them faster. And the reader of those docs is increasingly a coding agent, which will treat a stale page as current truth and write code to match it. Our context engineering field guide calls this a clash: the doc says one thing, the code says another, and the model has no way to know which one is authoritative.

The fix is to tie each page to its sources and re-run the source check whenever those sources change. If you already practice docs as code on GitHub, the pull request is the natural place for that check: a change to payments/retry.ts should surface every doc that cites it, in the same PR, before merge. Approving a generated page is not the end of its review. It's the start of a subscription.

Making the review cheaper#

Most of the work in the review pass is mechanical, and the parts that aren't get much faster when the generator does its share. Three things make the biggest difference.

Generate from the code with citations attached. A page that names the files it was written from turns step 1 into a glance and steps 3 and 4 into a short read of named files instead of a search. This is the single property worth selecting a generator on.

Check docs against changes at the moment of the change. The drift trap closes when the check runs in the pull request, not in a quarterly docs audit. That applies doubly when an agent wrote the change, since the agent has the changed files in context and can fix the affected doc before anyone reviews it.

Make updates reviewable diffs, not rewrites. When a doc does fall behind, the correction should arrive as a small, scoped change you can verify against the commit that caused it, not a regenerated page you have to re-review from the top.

That's the loop Moxie Docs is built around. It indexes your GitHub repository into living documentation with source citations attached to what it writes, so you can check a claim against the file it came from. It serves those docs, your conventions, and verified commands to coding agents over MCP, and its moxie.review_change tool lets an agent check a change before committing — including docs the change would make false — so the fix lands in the same PR. When docs fall behind the code anyway, it opens a reviewable Cleanup PR with the source-cited change; it proposes, you merge, never the reverse. The weekly Friday Cleanup is on the Pro and Team plans, and you can see what the first index finds on your own repository on the free plan: one repository, no card required. More on how the drift check works is on the documentation drift page.

Measuring whether it's working#

You don't need a quality program to know whether your review of AI-generated docs is working. A few cheap signals cover most of it.

• Mechanical failures per generated page. Missing paths, unknown identifiers, dead commands. If this number stays above zero, the generator isn't grounded well enough, and fixing that beats reviewing harder.
• Corrections after merge. Count edits to generated pages that fix a factual error within thirty days of approval. These are review misses, and their claim type tells you which step of the pass to tighten.
• Pages without source references. Each one is a page nobody can re-verify cheaply. Drive it toward zero.
• Behavior changes that shipped without a doc change. If code changes keep landing without touching the docs that cite them, you're accumulating documentation debt no matter how good the initial review was.
• Agent mistakes traced to a doc. When a coding agent writes the wrong pattern, check whether a stale doc told it to. This is the most expensive kind of doc error, and the one most worth counting.

None of these need a dashboard. A look at the last twenty generated pages and the last twenty merged PRs will tell you most of it.

Frequently asked questions#

Can AI generate documentation from source code accurately?#

Often, for the parts of the code that behave the way code like it usually behaves. Accuracy drops on unusual defaults, deliberate exceptions, error handling, and anything the model has to infer from names rather than read from implementation. Kang, Milliken, and Yoo found that roughly a fifth of the best model's generated comments contained demonstrably inaccurate statements, so generated docs should be treated as a strong first draft that requires verification, not as finished documentation.

How do you check AI-generated documentation for errors?#

Verify mechanically first, then by reading. Script the checks for file paths, identifiers, commands, and links, since those are cheap to automate and are a common source of invented details. Then trace each number and behavioral claim to the source file it describes, and look specifically for missing failure modes and permission checks. Read for clarity and tone last.

Should AI-generated documentation be labeled as AI-generated?#

That's a policy choice, but the more useful label is provenance: which source files a page describes and when it was last checked against them. A reader can act on "describes payments/retry.ts, last updated with the commit that changed it" in a way they can't act on "written by AI." Label the sources, and keep the label current.

Is AI-generated documentation good enough for coding agents to rely on?#

Only if it stays current. Agents read documentation as ground truth and can't tell a stale page from an accurate one, so an outdated generated doc is worse than no doc: it confidently points the agent at the wrong pattern. If you serve generated docs to agents, pair them with a check that flags pages whose cited source changed, and keep always-on files like AGENTS.md short enough to audit by hand.

Who should review AI-generated docs, a technical writer or an engineer?#

The engineer who owns the code should verify accuracy, because only someone who knows the system can catch a plausible but wrong claim. A technical writer adds the most value on structure, audience, and what's missing. If you have to pick one, pick the code owner. Prose problems are rarely what's wrong with generated docs.

The short version#

AI-generated documentation fails by being almost right: fluent, well structured, and wrong about the unusual parts that readers most need. Reviewing it well means verifying rather than reading. Script the checks for paths, identifiers, and commands; trace every number and behavior to a cited source; look hardest at what's missing; and treat clarity as the last question, not the first. Then close the gap that review can't cover: tie each page to the code it describes and re-check it when that code changes. A generated doc isn't trustworthy because someone approved it once. It stays trustworthy only while something keeps checking it against the code.