Skip to main content
← Blog
12 min readThe Moxie Docs team

Documentation Drift Detection: CI Checklist

Documentation drift detection starts with evidence: use this CI checklist to connect code changes to doc claims, triage findings, and review fixes.

  • documentation-drift
  • github
  • docs-as-code
  • ai-coding-agents
  • documentation-automation

A pull request changes DEFAULT_MAX_ATTEMPTS from five to three. Tests pass. The retry guide still promises five attempts, and the next coding agent copies that promise into a new worker. Documentation drift detection should catch that mismatch before the merge, with enough evidence for someone to fix it.

Build the check around claims and their sources. A changed file tells you where to look; it doesn't prove that a page is wrong. The checklist below separates confirmed contradictions from review candidates, then shows how to carry that distinction through CI, agent context, and documentation cleanup pull requests.

Triage Findings by Evidence#

Use this checklist as your initial CI policy. Treat the classifications as your team's explicit contract with reviewers, rather than presenting every automated finding as a defect.

FindingEvidence requiredInitial CI responseReviewer action
Broken referenceA documented repository path or symbol no longer existsFail a deterministic checkFix the reference or remove the obsolete claim
Failed exampleA documented command or example fails in its declared test environmentFail a deterministic checkCorrect the example or explain the environment requirement
Confirmed contradictionA specific claim conflicts with an authoritative value or behavior testBlock after the detector has earned trustReview the evidence and change the code or documentation
Possible semantic driftRelated source changed, but no contradiction is establishedReport an advisory findingConfirm whether the claim still holds
Missing coverageA new documented interface has no matching explanation or exampleReport an advisory findingDecide whether documentation is required

The distinction matters during ordinary refactors. Moving a retry helper may invalidate a source link without changing retry behavior. Changing a default may invalidate a sentence without breaking any links.

Start with advisory findings for uncertain cases. Promote a detector to a required check only when you can explain its failure condition and show how a reviewer resolves it. Keep an escape route for genuine exceptions, with a reason, an owner, and a point at which the exception must be reconsidered.

Map Claims to Their Sources#

A useful detection unit is a claim: “The worker makes three delivery attempts by default.” Attach it to the source that establishes the behavior, the page that describes it, and the environment where it applies.

Here's an illustrative repository layout:

Code
src/worker/retry_policy.py
tests/worker/test_retry_policy.py
deploy/worker-values.yaml
docs/operations/delivery-retries.md
AGENTS.md

The guide might contain separate claims about the default attempt count, retryable errors, backoff, and operational overrides. Those claims can depend on different files. Mapping the entire guide only to src/worker/ leaves the detector with a broad, noisy relationship.

Start with a small evidence manifest. This is an example schema you would implement in your own checker, not a standard configuration format:

YAML
claims:
  - id: worker.retry.default-attempts
    document: docs/operations/delivery-retries.md
    section: default-policy
    statement: The default maximum is three delivery attempts.
    sources:
      - path: src/worker/retry_policy.py
        symbol: DEFAULT_MAX_ATTEMPTS
      - path: tests/worker/test_retry_policy.py
        symbol: test_default_attempt_limit
    scope: application-default
    owner: worker-maintainers

Keep the statement narrow enough to check. “Retries are reliable” has no useful pass condition. “Authentication failures are never retried” can point to an error classifier and a behavioral test.

Record the commit examined in each detection result. Use commit-pinned source references for review evidence, and retain repository paths for finding the current source. A mutable branch link is useful for browsing, but your finding needs to establish which revision produced it.

You don't need to annotate every paragraph before starting. Pick the claims whose failure would send someone down the wrong operational path: setup commands, authentication requirements, configuration defaults, public interfaces, and destructive procedures. Leave historical explanations outside the first pass unless they assert current behavior.

Combine Three Detection Methods#

Use these methods together. Each answers a different question, so give their findings different confidence levels.

Reference Validation#

Reference validation checks whether documented paths, anchors, symbols, and configuration keys still resolve. Use it for precise assertions that can be evaluated without interpreting prose.

Best for: Renames, deleted files, removed options, and broken navigation.

For example, a guide points to src/worker/retries.py, but a refactor moves the implementation to src/worker/retry_policy.py. A path check can identify the broken reference. It cannot establish whether the guide's explanation of retry behavior remains correct.

I'd make narrowly defined reference failures required once the checker understands your repository's link conventions. The tradeoff is limited coverage: a perfectly valid link can lead to code that contradicts the surrounding sentence.

Executable Examples#

Executable examples run documented commands or assertions in a controlled environment. Use them where the documentation promises an observable result.

Best for: Developer onboarding documentation, CLI examples, request payloads, and configuration snippets.

For example, extract a setup command into a fixture that installs the locked dependencies and starts a disposable test service. Check the expected exit status and a meaningful result. A command that exits successfully after silently ignoring an obsolete option should still fail your documentation test.

I'd prioritize this method for the first successful local setup. Its tradeoff is maintenance: fixtures need declared dependencies, safe credentials, and predictable services. Never execute arbitrary documentation snippets against production just to check freshness.

Change-Triggered Semantic Review#

Change-triggered semantic review compares affected claims with relevant source changes. Use it for explanations that depend on control flow, relationships, or several files.

Best for: Architecture descriptions, failure handling, permission boundaries, and operational behavior.

For example, a pull request changes which errors enter the retry loop. The attempt-count example still passes, but the paragraph about retryable failures may now be wrong. A semantic review should identify that paragraph, cite the changed classifier, and explain the suspected contradiction.

I'd keep this advisory until its findings are consistently actionable. The tradeoff is uncertainty: source proximity and plausible language aren't proof. A semantic finding needs explicit evidence and a human decision.

Run Detection Around the Merge#

Use a pull request check for immediate feedback and a default-branch reconciliation pass for changes that escape the first check. The following is an illustrative workflow you can implement with your existing CI and repository scripts.

1. Establish the comparison.

Record the pull request's base and head commit identifiers. Ensure your checkout contains both commits and their common ancestor before collecting the proposed changes:

Shell
# BASE_SHA and HEAD_SHA come from trusted CI event metadata.
git diff --name-status --find-renames "$BASE_SHA...$HEAD_SHA"

Preserve rename and deletion information. A deleted source file deserves different handling from an edited implementation. Compare claims against the proposed head revision; retain the base revision to explain what changed.

2. Select affected claims.

Match changed files to the evidence manifest. Include changes to tests, deployment configuration, dependency manifests, and the manifest itself. Also check for newly added public interfaces that have no mapped documentation.

Selection should produce candidates, not automatic accusations. If a refactor touches a mapped file but preserves behavior, the correct outcome may be “reviewed, no documentation change required.”

3. Run checks in a stable order.

Start with reference validation, then executable examples, then semantic review. Give the semantic review the relevant diff, claim, source files, and test results. Don't ask it to inspect an entire repository when the finding concerns one configuration default.

For required GitHub Actions checks, avoid a workflow-level path filter that prevents the check from reporting at all. GitHub documents that workflows skipped by path or branch filtering can leave required checks pending. Run the workflow, then let its detection logic report that no relevant claims changed. See the official GitHub Actions workflow syntax documentation.

4. Report a finding someone can resolve.

Include the claim identifier, document location, source location, examined commits, evidence class, and proposed next action. Here's an example finding:

JSON
{
  "claim": "worker.retry.default-attempts",
  "classification": "confirmed-contradiction",
  "document": "docs/operations/delivery-retries.md#default-policy",
  "documented_value": 5,
  "source_value": 3,
  "source": "src/worker/retry_policy.py:DEFAULT_MAX_ATTEMPTS",
  "action": "Review the intended default and update the conflicting side."
}

Don't assume the code is always the side to preserve. The change may violate an intended contract. Documentation detection can expose that disagreement without deciding product behavior.

5. Reconcile after merge.

Run the checks against the resulting default-branch revision. Record the last successfully examined commit, and advance that checkpoint only after a complete run. If detection fails or times out, retain the unexamined range.

Keep the check read-only unless it explicitly proposes a separate documentation change. Any generated correction still needs human review before merge.

Catch the Self-Consistent Lie#

The non-obvious failure mode is a check that agrees with its own incomplete evidence.

Suppose DEFAULT_MAX_ATTEMPTS is three. A test confirms three. A generator updates the guide to say “Production workers make three delivery attempts.” Everything appears consistent.

But deploy/worker-values.yaml sets MAX_ATTEMPTS: 5 for production. The checked sources establish the application default, while the sentence promises effective production behavior. The detector has approved a claim broader than its evidence.

Treat scope as part of the claim. Separate an application default from an environment override, a supported interface from an internal helper, and intended behavior from an observed deployment. If production behavior depends on configuration outside the repository, say that the repository evidence is incomplete. Don't turn that gap into a confident sentence.

There's another version of this problem: the same extraction produces both the documentation and its expected test output. If both consume an incorrect field, they agree without independently testing the promised behavior. For consequential claims, add an assertion at the interface the reader actually uses.

Test your detector with deliberate documentation defects in a disposable branch. Change a documented value without changing code. Rename a referenced source. Introduce a configuration override. Confirm that each case produces the right classification, including “insufficient evidence” when appropriate.

The review pass still matters after detection. How to Review AI-Generated Documentation: Catching the Page That's Almost Right covers checking the proposed prose against its evidence.

Keep Agent Context in Scope#

Human-facing documentation explains behavior, rationale, and procedures. Agent context tells a coding agent how to work in the repository or gives it access to relevant evidence. Track both, but don't copy every explanation into every context file.

Use AGENTS.md for concise repository instructions when your chosen agent supports it: where tests live, which commands to run, which generated files to leave alone, and where authoritative guidance resides. An important AGENTS.md best practice is to link to a maintained explanation instead of duplicating a long operational claim.

Use CLAUDE.md for Claude Code repository instructions. Anthropic documents its automatic loading of these files as project context. That makes an obsolete test command a context defect worth checking, even when the human onboarding guide is correct. See the official Claude Code memory documentation.

Choose an MCP server for documentation when the agent needs to retrieve repository information during a task. MCP exposes capabilities such as tools and resources; the protocol supplies access, while your documentation process supplies trustworthy content. A live connection doesn't establish that an underlying claim is accurate. The official MCP server concepts documentation describes those building blocks.

Use llms.txt as a discovery aid when you publish a curated entry point for machine readers. Don't use it as the evidence ledger for private repository claims or assume that every agent reads it.

For detection, include instruction files and published discovery entries in the affected-document set. For retrieved context, ask which repository revision the answer represents. If the retrieval system can't establish that, keep the uncertainty visible during review.

Try Source-Backed Cleanup on One Repo#

Moxie Docs fits when you want source-backed documentation generated from a GitHub repository, updates as code changes, repository context over MCP, and proposed documentation fixes in reviewable pull requests.

The core documentation workflow and MCP context are available on every plan. Pro and Team add weekly Cleanup automation. Cleanup PRs are proposed changes: your team reviews and merges them. The plan comparison separates core access from weekly automation, and the free plan gives you a way to evaluate the workflow on one repository.

For an illustrative evaluation, choose a service with a documented configuration default and an onboarding command. Record the current source, make a small behavioral change in a branch, and inspect the resulting documentation workflow. Check whether you can trace a proposed correction to the relevant source and understand its scope without rereading the whole service.

Then inspect the correction as a normal engineering change. Does it preserve useful exceptions? Does it describe the default or the deployed configuration? Would you approve the same wording if a teammate wrote it? Keep your existing deterministic checks while you evaluate the source-backed review process.

The Short Version#

Start with one consequential claim and a detector you can deliberately break. Require precise evidence for blocking findings, keep semantic suspicions advisory, and record the revision and environment each check covers. Expand coverage after reviewers can resolve findings without investigating the entire repository. A small, trustworthy check is a better merge requirement than a broad warning everyone learns to dismiss.

FAQ#

Is a recent documentation edit evidence that drift is fixed?#

No. An edit timestamp shows activity. Verify that the relevant claim was checked against the relevant source revision. Formatting changes and unrelated corrections shouldn't reset that claim's verification record.

Should every code pull request change documentation?#

No. Internal changes can preserve documented behavior. Let the author or reviewer record that outcome for affected claims. Requiring a token documentation edit creates noise and doesn't establish accuracy.

Can AI decide whether documentation is stale?#

Use AI to propose claim-to-source relationships and identify possible contradictions. Require inspectable evidence for each finding. Keep human review explicit for ambiguous behavior, intended contracts, and proposed documentation corrections.

What should you measure first?#

Track confirmed findings, dismissed findings, unresolved findings, and defects discovered after merge. Review why dismissals and misses occurred. Avoid using the number of flagged pages as a success measure; it rewards a noisy detector.

Republish or cite this article

You're welcome to republish this piece in full or in part. We just ask that you credit the original with a link back. See our republishing guidelines.

Attribution snippet

<p>This article was originally published on <a href="https://moxiedocs.com/blog/documentation-drift-detection-ci-checklist">Moxie Docs</a>.</p>

Cite this article

The Moxie Docs team. "Documentation Drift Detection: CI Checklist." Moxie Docs, November 19, 2026, https://moxiedocs.com/blog/documentation-drift-detection-ci-checklist.

Try it on your repo

Put your own codebase on the same footing.

Searchable docs, MCP-ready context, and Cleanup PRs that keep everything current as the code changes.