Living Codebase Documentation: A Freshness Test
Test living codebase documentation with a claim-to-source map, merge checks, and a practical way to catch stale human docs and agent context.
- living-documentation
- documentation-drift
- github
- agent-context
- docs-as-code
A pull request changes a worker’s retry policy. The tests pass, the deployment succeeds, and docs/webhooks.md still tells the next engineer to expect the old behavior. Your coding agent reads that page and adds a test around the wrong retry count.
Living codebase documentation needs a freshness test: when a change makes a documented claim questionable, can you identify the claim, check its evidence, and get a correction reviewed? This guide gives you a concrete way to test that loop across human-facing pages and agent context.
Check Whether Your Docs Are Actually Living#
Use the following checklist on one recently changed subsystem. A page’s modification date tells you when someone edited it. These checks tell you whether you can trust its claims.
| Check | Passing evidence | Failure to investigate |
|---|---|---|
| Source connection | Important claims identify supporting files or tests | A broad repository link with no connection to specific behavior |
| Change coverage | A relevant code change identifies affected claims | Only changes inside docs/ trigger documentation checks |
| Revision alignment | The page or retrieved answer identifies the revision it describes | Branch code is explained with undocumented default-branch assumptions |
| Review boundary | Behavioral corrections arrive as inspectable diffs | Generated prose replaces an approved page without review |
| Reader outcome | A person or agent can complete a representative task correctly | The page builds, but its setup or operational instructions fail |
Start with claims that change engineering decisions: commands, configuration defaults, authorization rules, failure behavior, and service boundaries. Leave stylistic cleanup outside this test.
For the examples below, assume a GitHub repository with services/webhooks/, a shared configuration package, and docs/webhooks.md. The paths, configuration format, and procedures are illustrative. They describe a workflow you can implement; they aren’t a standard or a product configuration.
The useful distinction is between a claim that has been checked and a claim that merely survived the last merge. Give those states different names in your workflow.
Map Claims to Their Evidence#
A whole page is usually too large a unit for documentation drift detection. The retry count might change while the delivery format, signature verification, and operational guidance remain accurate. Regenerating the entire page makes the reviewer rediscover those boundaries.
Map a few consequential claims instead. Keep the map small enough that someone will maintain it.
1. Pick a reader decision.
Start with a sentence someone might act on: “Webhook delivery stops after the configured maximum number of attempts.” Record what would prove or disprove it. A function name alone is weak evidence; the implementation and a boundary test are better evidence.
2. Record the dependencies behind that decision.
In this example, the effective behavior depends on the retry loop, its configuration, and the tests. An illustrative registry could look like this:
# docs/claims.yml — illustrative, team-defined format
claims:
- id: webhook-delivery-attempts
page: docs/webhooks.md
section: delivery-failures
statement: Delivery stops at the configured attempt limit.
evidence:
- services/webhooks/src/retry.ts
- packages/config/src/webhooks.ts
- services/webhooks/test/retry.test.ts
owner: delivery-team
status: verified
checked_revision: REPLACE_WITH_FULL_COMMIT_SHA
Replace the revision placeholder when you actually check the claim. Don’t stamp the current commit onto every entry during a build; that would turn a verification record into a build record.
3. Capture reasoning separately from behavior.
The retry loop can establish what happens. It cannot establish why your team chose that policy. Put the decision and its rationale in an architecture decision record or a short explanation owned by the responsible engineer. An agent should report missing rationale rather than infer it from the implementation.
4. Treat the registry as a starting point.
File-level dependencies are coarse. Formatting a test may trigger an unnecessary check, while adding a new configuration override may escape the map entirely. Accept that limitation during the pilot. Expand the evidence map when you find a missed dependency, rather than mapping every function before you have a working loop.
Give Each Reader the Right Surface#
Human-facing documentation explains tasks, behavior, and reasoning. Agent context tells an agent how to work and where to retrieve evidence. Keep those responsibilities clear, even when both readers use the same underlying pages.
Human-Facing Documentation#
This is the material an engineer reads to understand a subsystem or complete a task. Use it when sequence, explanation, alternatives, or operational consequences matter.
Best for: Onboarding guides, architecture explanations, and procedures that need human judgment.
For example, docs/webhooks.md can explain delivery guarantees, show how to reproduce a failure locally, and distinguish retryable failures from permanent failures. Developer onboarding documentation should link to that explanation from the first relevant task. Don’t require a new engineer to reconstruct the system from agent instructions.
AGENTS.md#
AGENTS.md is an open Markdown format for repository instructions aimed at coding agents. The official AGENTS.md guide describes uses such as setup commands, testing instructions, code style, and project-specific guidance. It fits repositories where your chosen agents support that convention.
Best for: Concise, shared instructions about how an agent should work in the repository.
For example, tell an agent to run the webhook tests after changing retry behavior and to inspect docs/webhooks.md for affected claims. Keep the retry-policy explanation in the page. My default is to put durable working rules in AGENTS.md and keep frequently changing subsystem facts elsewhere.
CLAUDE.md#
CLAUDE.md supplies persistent instructions to Claude Code. The Claude Code documentation describes instruction files as context rather than enforced configuration. Use it when your Claude Code workflow needs explicitly maintained instructions, and check which instruction files your installed setup loads.
Best for: Claude Code-specific guidance that your team deliberately maintains.
For example, record how Claude Code should verify a documentation change in this repository. Don’t create a second copy of every shared convention just because another filename is available. If you retain both instruction files, define which content belongs in each and test that arrangement in a fresh session.
MCP Repository Context#
For this workflow, an MCP server for documentation is a retrieval path: the agent requests relevant repository context when it needs it. Choose this approach when copying detailed subsystem knowledge into startup instructions would create too much duplicated material.
Best for: Task-specific access to source-backed documentation and repository evidence.
For example, before editing retry behavior, have the agent retrieve the delivery-policy explanation and its supporting files. Require the retrieval workflow to expose the repository and revision where possible. The tradeoff is another dependency: if retrieval fails or its revision is unclear, the agent must inspect the checked-out source rather than silently trust an old answer.
llms.txt#
The llms.txt proposal describes a Markdown overview that helps agents find and use website content. It fits a published documentation site where you want to expose a curated route into useful pages.
Best for: Discovery of published documentation, especially when readable Markdown versions are available.
For example, a public SDK documentation site might point readers toward authentication and migration guides. I wouldn’t choose llms.txt as the freshness mechanism for a private working branch. You still need to establish which code revision the linked documentation describes.
Turn Merges Into Targeted Rechecks#
Once claims have evidence, a merge becomes an invalidation event. A changed dependency means “check this claim again.” It doesn’t automatically mean the claim is false.
If you already have a docs-as-code workflow on GitHub, keep it. Add claim invalidation to the existing review path rather than creating a parallel documentation process.
1. Compare the revisions you intend to evaluate.
For an illustrative pull request check, compare the branch with its merge base against the target branch. Here, origin/main is the example target:
git fetch origin main
BASE=$(git merge-base origin/main HEAD)
git diff --name-status "$BASE" HEAD -- \
services/webhooks/ \
packages/config/ \
docs/
Your CI checkout must include enough history to find that merge base. For a post-merge check, supply the before-and-after revisions of the target branch. Don’t assume the previous commit always represents the complete change you want to inspect.
2. Match changed evidence to claims.
A change to packages/config/src/webhooks.ts should select the retry-policy claim even if no Markdown changed. Include deletions and renames in this comparison. A missing source file is useful evidence that a reference needs attention.
3. Classify the result before requesting an edit.
Use states such as verified, needs-review, and contradicted. A file change can move a claim into needs-review; a reproducible mismatch can justify contradicted. A private helper rename might require no prose change after inspection.
4. Propose the smallest correction.
Suppose the example application changes from five total attempts to three. The proposed documentation diff should change that statement and any dependent example. It should identify the relevant implementation and test. Rewriting the introduction, architecture section, and setup guide adds review work without addressing the mismatch.
Run deterministic checks where possible. Keep semantic judgment with the engineer who understands the behavior and its consequences.
Catch the Two-Revisions Failure Mode#
The non-obvious failure is a correct answer about the wrong revision.
Imagine an agent working on a feature branch that removes synchronous delivery. Its retrieved documentation accurately describes main, where synchronous delivery still exists. Every cited file is real. The explanation is internally consistent. The recommendation is still wrong for the working tree.
A “last updated” timestamp doesn’t resolve that mismatch. Neither does a successful retrieval request.
Give each answer a scope: repository, branch or commit, and relevant evidence. If your retrieval system cannot provide that information, treat its answer as orientation and verify behavior against the checked-out files before editing.
There’s a related release boundary. A merge can change the next release without changing the version currently deployed. An operational page may need to describe the deployed release; a contributor guide may need to describe the development branch. State that scope on the page when the difference matters.
Also watch for environmental overrides. In the example repository, retry.ts might supply a default that deployment configuration replaces. A claim that cites only the default can look well grounded while describing no production environment accurately. Include the override in the evidence map, or narrow the claim to the default explicitly.
Finally, source-backed documentation describes observed implementation behavior. If an implementation change violates an approved decision, automatically updating the docs could conceal the regression. Flag the disagreement for review. Don’t let a documentation refresh decide whether code or policy is authoritative.
Run a Controlled Freshness Test#
Prove the loop on a disposable branch before you roll it across every repository. Use a behavioral change whose documentation impact you can predict.
1. Establish the baseline.
Choose one page, one claim, and its evidence. Confirm the current statement against the implementation and tests. Record the checked revision and the expected owner.
2. Introduce a deliberate mismatch.
In the example repository, change the attempt limit and update its test, but leave the page untouched. This is a test fixture, not a change to merge into production. Keep deployments disabled for the experiment.
3. Inspect what the workflow selects.
It should identify the affected claim without requiring a Markdown edit. Check whether it names the correct page, supporting files, and changed behavior. A generic “documentation may need updating” comment is a weak result.
4. Test the negative case.
Make a separate internal refactor that preserves behavior. The workflow may request a recheck, but it shouldn’t demand a fabricated prose change. Repeated false alarms will teach reviewers to ignore the signal.
5. Check both readers.
Have an engineer follow the affected instruction. In a fresh agent session, ask for an explanation of the behavior and the evidence behind it. Check the revision as well as the answer. A correct page and a stale retrieval path are separate defects.
6. Close the loop through review.
Inspect the proposed correction, approve or reject it, and verify what readers receive afterward. Record missed claims, unnecessary edits, and revision mismatches. Those observations tell you where to improve the map.
Use Moxie Docs for the Maintenance Loop#
Moxie Docs generates source-backed documentation from GitHub repositories, updates it as code changes, provides repository context over MCP, and proposes documentation fixes in reviewable pull requests. That fits the maintenance work after you’ve identified which explanations your team needs.
Keep the plan distinction clear. Every plan includes the MCP context endpoint. Weekly Friday Cleanup automation is on Pro and Team; don’t assume that weekly schedule applies to Free or Starter. The plan comparison separates those capabilities. Cleanup PRs are proposed changes for your team to review and merge. They aren’t automatic merges.
For an illustrative evaluation, connect the webhook repository on the free plan, inspect a generated explanation against its cited source, and compare the available MCP context with your working checkout. Inspect any proposed documentation correction through your normal review process. Use the controlled freshness test above to evaluate the workflow; don’t assume a tool will resolve policy conflicts or deployment scope for you.
Start with a subsystem whose behavior you understand. You need that baseline to judge whether the evidence, explanation, and proposed correction agree.
The Short Version#
Choose one behavior-changing pull request and trace a consequential claim through its human-facing page, agent instructions, and retrieved context. Record the evidence and revision, deliberately create a mismatch on a disposable branch, and check whether the workflow identifies the right claim without rewriting unrelated material. Keep human review at the correction boundary. Expand only after that small test catches both stale prose and correct answers drawn from the wrong revision.
FAQ#
How Is Living Codebase Documentation Different From Docs as Code?#
Docs as code gives documentation version control, checks, and review. Use living codebase documentation to describe the additional maintenance loop: source changes identify affected claims, those claims are rechecked, and corrections return through review.
Can Documentation Drift Detection Be Fully Automated?#
Automate concrete checks such as missing paths, executable examples, and known configuration values where practical. Treat behavioral mismatches as evidence for review. Rationale, release scope, and disagreements between implementation and policy need engineering judgment.
Should AGENTS.md Contain the Whole Architecture?#
My default is no. Put working rules and routes to relevant evidence in AGENTS.md. Keep detailed architecture explanations in human-facing pages that agents can also read. This reduces duplicated material you must invalidate after changes.
What Belongs in Documentation Cleanup Pull Requests?#
Keep each correction narrow: the affected claim, its supporting source change, and necessary example updates. Separate factual corrections from broad prose cleanup so reviewers can verify behavior without sorting through unrelated edits.
Republish or cite this article
You're welcome to republish this piece in full or in part. We just ask that you credit the original with a link back. See our republishing guidelines.
Attribution snippet
<p>This article was originally published on <a href="https://moxiedocs.com/blog/living-codebase-documentation-freshness-test">Moxie Docs</a>.</p>Cite this article
The Moxie Docs team. "Living Codebase Documentation: A Freshness Test." Moxie Docs, November 17, 2026, https://moxiedocs.com/blog/living-codebase-documentation-freshness-test.
Read next
How to Review AI-Generated Documentation: Catching the Page That's Almost Right
AI-generated documentation reads fluently whether it's true or not. A practical review pass for docs generated from code: the failure modes worth checking, the claims to verify first, scripts that catch the mechanical errors, and how to stop correct docs from going wrong after you approve them.
How We Dogfood a Continuous Documentation Workflow to Keep Code, Docs, and AI in Sync
Inside Moxie Docs' own continuous documentation workflow: how we keep code, docs, and AI agent context in sync automatically, and the setup that speeds onboarding.