Incident Response Checklist
An incident response checklist for small IT and ops teams: declare, assign roles, mitigate, communicate and hand over cleanly. Start it from your alerting.
- Typical schedule per incident
- The checklist 5 sections, 22 items
- Variables
{{incident_title}}{{severity}} - Start from CI
Use this checklist the moment someone suspects that a production service is down, degraded or losing data. It is written for small teams without a dedicated incident desk, where the same few engineers both fix the problem and talk to customers. The goal is not to slow the fix down. It is to make sure that the obvious things happen in the first ten minutes, even at 3 a.m.
Who owns it
The incident commander owns the run. In a small team this is often the first responder, but the role should pass to someone else as soon as a second person joins, so the person debugging can focus. The commander decides severity, assigns roles and calls the incident resolved.
Common pitfalls
- Everyone debugging, nobody coordinating. Name an incident commander early, even if it is you. Coordination falls apart when five people investigate in parallel and nobody updates customers.
- Fixing the root cause during the outage. Mitigate first: roll back, fail over, scale up or switch a feature flag off. Root-cause work belongs in the postmortem.
- Silent channels. Stakeholders assume the worst when there are no updates. Post on a fixed rhythm, even if the update is "still investigating".
- No timeline. Memory fades within hours. Note times and actions in the incident channel as you go.
Start the run from your alerting tool, so the checklist and its timestamps exist before anyone opens a laptop. Each check-off is then part of the audit trail for the postmortem.
The checklist: Incident response: {{incident_title}}
Declare
-
Impact confirmed: which service, which users, since when
Check from the outside (status checks, a real login), not only from internal dashboards.
-
Severity set to
{{severity}}using your severity definitionsIf unsure, pick the higher severity. Downgrading later is cheap; escalating late is not.
-
Incident channel or bridge opened and linked in this run
-
Incident commander named and announced in the channel
-
Communications owner named optional
Needed for anything customer-visible; for a small internal incident the commander can do both.
Stabilise
-
Recent changes checked: deploys, config changes, infrastructure work, expiring certificates
Most incidents follow a change. Your deploy log and change calendar are the first place to look.
-
Mitigation chosen and recorded: rollback, failover, feature flag off, scale up or traffic block
-
Mitigation applied and its effect checked on the same metric that showed the problem
-
Data integrity checked: no corrupt writes, lost jobs or duplicate payments optional
Relevant whenever databases, queues or payment flows were involved.
-
Escalated to vendor or cloud provider support with a ticket number optional
Communicate
-
Internal stakeholders informed: support, account managers, leadership for high severities
-
Status page incident opened with plain-language impact
-
Next update time stated in every update, and kept
-
Affected key customers contacted directly optional
Resolve
-
Service confirmed healthy for an agreed period after mitigation
Watch error rate, latency and queue depth, not only the health check.
-
Temporary measures listed: disabled features, manual workarounds, extra capacity
-
Status page and internal channel updated to resolved with a short summary
-
Incident end time recorded
Hand over
-
Timeline of key events and decisions copied from the channel into the incident record
-
Postmortem owner and date agreed
-
Follow-up tickets created for temporary measures that must be reverted
-
On-call engineer briefed if the incident spans a shift change
Checklist as Markdown
Start this checklist from your pipeline
From an alert webhook, POST /api/v1/runs with template "Incident response: {{incident_title}}" and the alert's severity as severity.
curl -X POST https://runbook.metastic.ch/api/v1/runs \
-H "Authorization: Bearer $RUNBOOK_TOKEN" \
-H "Content-Type: application/json" \
-d '{"template": "Incident response: {{incident_title}}", "variables": {"incident_title": "…", "severity": "…"}}'Frequently asked questions
What are the main steps of incident response?
For an operational outage, the practical steps are: declare the incident and set severity, assign roles, mitigate the impact, communicate on a fixed rhythm, confirm recovery and hand over to a postmortem. Security incident frameworks add containment, eradication and evidence preservation, which you should plan separately with your security lead.
Who should be the incident commander?
Anyone on the team who is trained for the role and is not the main person debugging. The commander coordinates, decides and communicates, and does not need to be the most senior engineer. Small teams often rotate the role with the on-call schedule.
How often should we post status updates during an outage?
Pick a rhythm that matches the severity, for example every 30 minutes for a major outage, and state the next update time in each message. Regular updates with no news build more trust than long silences followed by a detailed report.
Can I start this checklist automatically when an alert fires?
Yes. Configure your alerting tool (e.g. Alertmanager, Grafana or PagerDuty) to call POST /api/v1/runs
with the template name and the variables incident_title and severity. The run then exists with a
timestamp from the first minute, and responders check off steps as they go.
Updated