Skip to main content
Moxie Docs

FREE TOOL

Runbook Template Generator

A free DevOps runbook template you can fill in, preview, and download as Markdown.

Start from a sample runbook for API errors, a database failover, or certificate expiry, then replace the details with your own service. No account needed.

Read how to write a runbook

A runbook is only useful if it matches the system on call tonight Export free above · connect a repo when you want Moxie Docs to keep it accurate

See how Moxie Docs keeps docs currentConnect GitHub free
Incident-ready structure

Trigger, impact, diagnosis, mitigation, rollback, verification, and escalation, in the order a responder needs them.

Markdown export

Copy or download a runbook.md that renders in GitHub, GitLab, Confluence, Notion, and most wikis.

Shareable drafts

The draft is encoded in the link, so you can send it to a teammate for review without an account.

Runs in your browser

Nothing you type is uploaded. Hostnames, commands, and dashboards stay on your machine.

Runbook examples

Three complete sample runbooks built with the template. Load any of them in the editor above and adapt it to your service.

Sample runbook: Checkout API: elevated 5xx errors

Runbook: Checkout API: elevated 5xx errors

Service#

  • Service: checkout-api
  • Owner: Payments team
  • On-call: #payments-oncall (PagerDuty: Payments Primary)

When to use this runbook#

  • Alert: CheckoutApi5xxRateHigh fires (5xx > 2% of requests for 5 minutes)
  • Customers report failed payments in #support-escalations

Customer impact#

Customers cannot complete purchases. Treat as SEV-2 if more than 5% of checkouts fail, SEV-1 above 25%.

Before you start#

  • Grafana: Checkout API overview dashboard
  • Logs: service:checkout-api status:5xx in the log explorer
  • Deploy history: #deploys channel and the CD pipeline
  • Stripe status page and the Stripe dashboard

Diagnose#

  1. Check whether a deploy shipped in the last 30 minutes.
  2. Split 5xx by route: is it all routes or only POST /payments?
  3. Check Stripe API error rate and latency on the dashboard.
  4. Check database connection pool saturation and slow queries.
  5. Look for a single noisy customer or a traffic spike.

Useful commands#

Shell
kubectl -n payments get pods -l app=checkout-api
kubectl -n payments logs deploy/checkout-api --since=15m | grep ERROR
kubectl -n payments rollout history deploy/checkout-api

Mitigate#

  1. If a deploy caused it, roll back to the previous release.
  2. If Stripe is degraded, enable the checkout-queue feature flag so payments retry asynchronously.
  3. If the connection pool is saturated, scale checkout-api to 2x replicas.
  4. If one customer is responsible, rate-limit their API key.

Roll back#

  • kubectl -n payments rollout undo deploy/checkout-api
  • Disable the checkout-queue flag once Stripe recovers.

Verify recovery#

  • 5xx rate under 0.5% for 15 minutes.
  • Successful checkouts per minute back within 10% of the same time last week.
  • No new errors in #support-escalations.

Escalate#

  • No improvement after 20 minutes: page the Payments secondary.
  • SEV-1 or data integrity risk: page the incident commander.
  • Stripe-side outage: open a Stripe support case and post in #status.

After the incident#

  • Write the incident timeline within 24 hours.
  • Schedule a blameless postmortem for SEV-1 and SEV-2.
  • Update this runbook with anything that was missing or wrong.

Sample runbook: Postgres primary failover

Runbook: Postgres primary failover

Service#

  • Service: orders-db (Postgres 16)
  • Owner: Platform team
  • On-call: #platform-oncall

When to use this runbook#

  • Alert: PostgresPrimaryDown or replication lag > 60s
  • Application logs show connection refused to the primary

Customer impact#

Writes fail across orders and accounts. SEV-1 until writes recover.

Before you start#

  • Database dashboard: replication lag, connections, disk
  • Cloud console access with the db-operator role
  • Connection string secret in the secrets manager

Diagnose#

  1. Confirm the primary is unreachable from two different hosts.
  2. Check replication lag on each replica.
  3. Check disk space and recent maintenance events in the cloud console.

Useful commands#

Shell
psql "$REPLICA_URL" -c "select now() - pg_last_xact_replay_timestamp() as lag;"
psql "$PRIMARY_URL" -c "select 1;"

Mitigate#

  1. Pick the replica with the lowest lag.
  2. Promote it with the managed failover action, or pg_ctl promote on self-hosted.
  3. Point the application connection secret at the new primary and restart the app pods.

Roll back#

  • Do not fail back automatically. Rebuild the old primary as a replica once it is healthy.

Verify recovery#

  • Writes succeed from the application.
  • Error rate back to baseline.
  • A new replica is streaming from the promoted primary.

Escalate#

  • Failover not complete in 15 minutes: page the database lead.
  • Any sign of data loss: page the incident commander and freeze deploys.

After the incident#

  • Record the data-loss window, if any.
  • Re-establish two replicas.
  • Review why the primary failed.

Sample runbook: TLS certificate about to expire

Runbook: TLS certificate about to expire

Service#

  • Service: public edge (api.example.com, app.example.com)
  • Owner: Infrastructure team

When to use this runbook#

  • Alert: CertificateExpiresSoon (< 14 days)
  • Browser or client TLS errors after expiry

Customer impact#

After expiry, every client connection fails. Before expiry, no impact.

Before you start#

  • DNS provider console
  • Certificate manager or ACME client host
  • CDN or load balancer TLS settings

Diagnose#

  1. Check which hostnames the certificate covers and its exact expiry.
  2. Check whether automatic renewal ran and why it failed.

Useful commands#

Shell
echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null | openssl x509 -noout -dates -subject

Mitigate#

  1. Fix the renewal failure (DNS challenge record, rate limit, expired credentials).
  2. Trigger a manual renewal.
  3. Upload the new certificate to the CDN or load balancer if it is not automatic.

Roll back#

  • Keep the previous certificate available until the new one serves on every hostname.

Verify recovery#

  • openssl shows the new expiry date on every hostname.
  • No TLS errors in client logs.

Escalate#

  • Under 48 hours to expiry and renewal still failing: page the infrastructure lead.

After the incident#

  • Add monitoring for the renewal job itself, not only for the expiry date.

Runbook template FAQ

What is a runbook?

A runbook is a step-by-step procedure for handling a specific operational event, such as an alert firing, a database failover, or an expiring certificate. It tells the on-call engineer how to confirm the problem, the safest way to mitigate it, how to check that it is fixed, and who to escalate to.

What should a runbook template include?

At minimum: the alert or symptom that triggers it, customer impact and severity, the access and dashboards needed, ordered diagnosis steps, mitigation steps with the safest fix first, how to roll back, how to verify recovery, escalation rules, and follow-up tasks after the incident.

What is the difference between a runbook and a playbook?

A runbook covers one concrete procedure with exact commands. A playbook is broader: it describes how a team responds to a class of incidents, including roles, communication, and decision-making, and usually links to several runbooks.

Where should runbooks live?

Keep them where responders already look during an incident, ideally next to the code in the repository (for example docs/runbooks/) and linked from the alert itself. Runbooks in the repo can be reviewed in pull requests and updated when the service changes.

How do I keep runbooks from going stale?

Update the runbook in the same pull request that changes the service, review runbooks after every incident, and link each alert to its runbook so broken links surface quickly. Moxie Docs flags documentation that a merged change has made stale and opens a pull request to fix it.

Keep it true

A runbook is only useful if it matches the system on call tonight

Draft and export your runbook here. Connect GitHub when you want Moxie Docs to flag runbooks and docs that a merged change made stale, and open a pull request to fix them.

See how Moxie Docs keeps docs currentConnect GitHub free

Export free above · connect a repo when you want Moxie Docs to keep it accurate