How to Write a Runbook: A Field Guide for On-Call Engineers Who Will Read It at 3 a.m.

The best time to learn how to write a runbook is not during an incident, but that is when most teams find out theirs was wrong. The page fires, the on-call engineer opens the linked doc, and step two runs a script that was deleted in March. Now they are debugging the outage and the runbook, with the incident channel watching.

The case for runbooks is strong. In the chapter on being on-call, Google's SRE book points out that stress hormones push responders toward "unreflective and unconsidered (but immediate) action." The book's introduction adds a number: writing best practices down ahead of time in a playbook "produces roughly a 3x improvement in MTTR as compared to the strategy of 'winging it.'" That gain only holds if the playbook is still true when someone opens it.

This guide covers what a runbook is and isn't, a structure that works when the reader is half-awake, how to write each step so it can't be misread, and the failure mode that tidy formatting won't fix: runbooks drift from the system they describe faster than almost any other doc you own.

What a runbook is, and what it isn't#

A runbook is a set of written steps for one specific operational situation, usually tied to one alert. It answers one question: this just happened, what do I do now? It isn't architecture documentation, a postmortem, or a tour of the service. Mixing those together is the most common reason runbooks get too long to use.

The words runbook and playbook get swapped all the time, and the SRE book itself says "playbook" for what most teams call runbooks. The label matters less than the scope. If a doc tries to answer two rows of that table, split it and link the two.

The anatomy of a runbook that works under pressure#

The reader is an engineer who just woke up, may never have touched this service, and is being asked for status updates every ten minutes. Every structural choice should cut down how much they have to think before acting.

flowchart TD
  A["Alert fires"] --> B["Open runbook linked from the alert"]
  B --> C["Confirm: is the symptom real?"]
  C -->|"No"| D["Silence, note false positive, file ticket"]
  C -->|"Yes"| E["Assess impact and severity"]
  E --> F["Mitigate with the listed steps"]
  F --> G{"Did it recover?"}
  G -->|"Yes"| H["Verify, hand off, update the runbook"]
  G -->|"No"| I["Escalate to the named owner"]

That flow suggests a fixed section order. Use the same order in every runbook so responders always know where to look:

1. Summary. One sentence on what the alert means in plain words.
2. Impact. What users experience while this is firing, and how to judge severity.
3. Triage. How to confirm the alert is real, with the exact query or dashboard.
4. Mitigation. Numbered steps that stop the bleeding. These come before diagnosis.
5. Verification. How you know it worked.
6. Escalation. Who to page next, by rotation name, and when.
7. Context. Links to the architecture doc, past incidents, and the source of the alert.

Mitigation comes before root cause on purpose. Responders want to understand the problem, and that's the right instinct for a postmortem. During the incident, the job is to get users unblocked. A runbook that opens with three paragraphs of theory teaches people to skip runbooks.

How to write a runbook, step by step#

1. Start from the alert, not the service. Write one runbook per alert, or per tight group of alerts with the same response. "Payments runbook" is too broad to act on. "payments-api p99 latency above 2s for 5 minutes" is something you can write steps for. Put the runbook URL in the alert definition so nobody has to search.

2. Write the summary for someone who has never seen the service. "The checkout API is responding slowly; customers may see spinners or timeouts at payment." No internal codenames without a gloss. Your most experienced engineer won't be on call forever.

3. Make triage concrete. "Check the dashboard" is not a step. Name the dashboard, link it, and say what bad looks like. Give the exact query to paste:

kubectl -n payments get pods -l app=payments-api \
  --field-selector=status.phase!=Running

If a healthy result and an unhealthy result look different, show both. People under stress match patterns. Give them the pattern.

4. Write mitigation steps as commands, not intentions. "Restart the workers" turns into an argument at 3 a.m. about which workers, how, and whether that drops in-flight jobs. Write the command, say what it does, and say what to expect:

kubectl -n payments rollout restart deployment/payments-worker
kubectl -n payments rollout status deployment/payments-worker --timeout=5m

Expected: all pods report Running within about three minutes. In-flight jobs are retried from the queue, so this is safe to run during business hours.

5. Mark every dangerous step. If a step deletes data, fails over a database, or pages another team, bold it and state the blast radius in the same sentence. Put an explicit decision point in front of it: "Only continue if error rate is still above 5% after step 3."

6. Name owners by rotation, not by person. "Ping Sam" breaks the day Sam changes teams. "Page payments-secondary in your incident tool" survives reorgs.

7. End with verification and a feedback hook. Tell the responder how to confirm recovery, then ask them to fix anything in the runbook that was wrong. That last line is how runbooks improve. Without it, they only decay.

A runbook template you can copy#

Keep runbooks in the repository next to the code they operate. A docs-as-code workflow gives you review, history, and a natural place to update them in the same PR that changes the system.

# Runbook: payments-api high p99 latency

**Alert:** `PaymentsApiLatencyHigh` (p99 > 2s for 5m)
**Owner rotation:** payments-primary · **Escalation:** payments-secondary
**Last verified:** 2026-10-13 against commit `a1b2c3d`

## Summary
Checkout requests are slow. Customers may see timeouts at payment.

## Impact
- Sev 2 if checkout success rate drops below 98%.
- Sev 1 if it drops below 90% or lasts more than 15 minutes.

## Triage
1. Open the "Payments / Latency" dashboard. Confirm p99 is elevated across all regions.
2. Check for unhealthy pods:
   `kubectl -n payments get pods -l app=payments-api`

## Mitigation
1. If a deploy went out in the last 30 minutes, roll it back:
   `kubectl -n payments rollout undo deployment/payments-api`
2. If the DB connection pool is saturated, restart workers (safe, jobs retry):
   `kubectl -n payments rollout restart deployment/payments-worker`
3. **Danger:** only if steps 1–2 fail, fail over the read replica. Affects reporting jobs.

## Verification
p99 under 800ms for 10 minutes and checkout success rate above 99.5%.

## Escalation
Page payments-secondary after 20 minutes without recovery.

## Context
- Architecture: docs/payments/architecture.md
- Past incidents: INC-412, INC-587
- Alert source: monitoring/alerts/payments.yaml

The Last verified line does more work than anything else in the template. It turns "is this still right?" from a feeling into a date and a commit you can diff against. If you want a visual of the escalation path or service dependencies, sketch it in the Mermaid diagram editor and paste the code block into the Context section.

Common runbook mistakes#

Most runbook problems are easy to spot once you know what to look for. The table pairs each symptom with a fix.

The last row is the hardest one to fix, and it gets its own section.

The trap: runbooks rot faster than any other doc#

Runbooks are full of the most change-prone facts in your system: deployment names, script paths, CLI flags, env var names, dashboard URLs, queue names. A README written as prose can stay roughly correct through a refactor. A runbook with scripts/drain-queue.sh --force in step three is wrong the moment someone renames the script.

Usage makes it worse. Most docs get some reads every week, so errors get noticed. A runbook for a rare failure might go a year unread. Then the one time it matters, it's documentation drift at the worst possible moment: during an outage, in front of a responder who can't tell which step is stale. The SRE book's 3x MTTR gain turns into a loss, because a wrong runbook is worse than none. It sends people confidently down the wrong path.

Reading the runbook more carefully won't catch this. The text is clear. It's just describing a system that no longer exists. You can only catch it by checking the runbook against the code, and that check has to happen when the code changes, not when the alert fires. This is the same mechanism behind documentation debt: the gap grows quietly, one reasonable PR at a time.

Three habits close most of the gap:

• Update runbooks in the PR that changes the system. If a PR renames a deployment, the reviewer should ask which runbooks mention it. A line in your pull request description template ("Runbooks affected:") makes this a reflex.
• Run game days. The SRE book's "Wheel of Misfortune" exercise has engineers walk through a past incident using the current docs. Stale steps show up quickly when someone actually tries to follow them.
• Use it, then fix it. Every real incident should end with the responder correcting whatever was wrong. Put that in your incident close-out checklist.

Keeping runbooks honest automatically#

Habits fix most of it, but they rely on someone remembering, and the PR that renames a script is rarely the one where anyone thinks about the incident doc three folders away. This is where tooling that reads the code and the docs together helps.

Moxie Docs indexes your GitHub repository into living documentation that cites its sources, and notices when code changes leave the docs behind. When that happens, it opens a Cleanup PR with the proposed fix for your team to review. It never merges anything on its own. It also serves your conventions, docs, and verified commands to coding agents through its MCP server, so an agent changing a deployment name can call moxie.propose_doc_update and include the runbook fix in the same PR instead of leaving it for the next incident. You can try it on one repository with the free plan, no card required.

Measuring whether your runbooks are working#

You don't need a program to know whether runbooks are helping. A few cheap signals tell you:

• Alert coverage. What share of paging alerts link to a runbook? Anything under 100% for paging alerts is a gap.
• Staleness. How many runbooks have a "last verified" date older than two quarters, or older than the last change to the code they describe?
• Postmortem mentions. How often does "the runbook was wrong or missing" show up as a contributing factor?
• Escalation rate. How often does the primary on-call escalate on alerts that have a runbook? A high rate on a documented alert means the runbook isn't doing its job.
• Edits after incidents. Did the runbook change after the last time it was used? If not, either it was perfect or nobody closed the loop.

None of these need a dashboard. A spreadsheet updated monthly will do.

Frequently asked questions#

What is the difference between a runbook and a playbook?#

In most teams, a runbook covers one specific operational scenario, usually tied to one alert, with concrete steps to triage and mitigate it. A playbook describes the broader incident response process: roles, communication, severity levels, and how to coordinate. Some organizations, including Google SRE, say "playbook" for alert-level docs too, so agree on terms within your team and keep each document's scope clear.

What should a runbook template include?#

At minimum: the alert it covers, an owning rotation, a one-sentence summary, user impact and severity guidance, triage steps with exact queries, numbered mitigation steps with commands, a verification check, an escalation path, and a "last verified" date. The template above includes all of these and can be adapted to any service.

Where should runbooks live?#

Ideally in the same repository as the service they describe, linked directly from the alert definition. Keeping them next to the code means the PR that changes the system can update the runbook in the same review, and it keeps runbooks available to the coding agents and tools that already read your repo.

How long should a runbook be?#

Long enough that someone who has never touched the service can mitigate the problem, and short enough to fit on roughly one screen per scenario. If a runbook keeps growing, it's probably covering several alerts or mixing in architecture explanation. Split it and link out to the deeper docs.

Should runbooks be automated?#

Steps that are safe, repeatable, and run the same way every time are good candidates for automation, and the runbook should then point to the automated job. Keep a written runbook anyway. Automation fails too, and the responder needs to know what the job does and what to do when it doesn't work.

The short version#

A runbook is a promise to someone who will be tired, stressed, and new to your service. Write one per alert, put mitigation before diagnosis, turn every step into a command with an expected result, and name owners by rotation. Then treat drift as the real enemy: stamp each runbook with a verified date, update it in the PR that changes the system, and fix it every time someone uses it. A runbook that was right last year can be wrong tonight.