Learn more about Well-Architected Operational Excellence → Incident Management
Incident Detection and Alerting
Patterns
| Where to look | What good looks like |
|---|---|
| Platform | Monitoring | ✅ Alerting architecture includes severity classification routing critical incidents (production unavailable, data corruption, security compromise) to page on-call immediately |
| Platform | Monitoring | ✅ High severity incidents (degraded performance, partial outage) generate immediate notifications for prompt business-hours investigation |
| Platform | Monitoring | ✅ Warnings (approaching thresholds, configuration drift) aggregate into daily digest for proactive remediation |
| Platform | Monitoring | ✅ Alert enrichment includes diagnostic context: affected user count, error rate percentiles, recent deployments, related alerts, and direct dashboard links |
| Platform | Monitoring | ✅ Alert context reduces mean time to acknowledge by providing immediate situation assessment without gathering basic information through separate queries |
| Platform | Monitoring | ✅ Alert grouping prevents alert storms by consolidating hundreds of related alerts from single underlying problem into one notification with aggregate statistics |
| Platform | Monitoring | ✅ Grouped alerts show count, rate, and affected transactions revealing problem pattern without overwhelming responders |
| Platform | Monitoring | ✅ Alert suppression prevents known-transient conditions from generating alerts during expected maintenance windows or auto-remediation periods |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Platform | Monitoring | ⚠️ Integration failure generates 100s individual alerts in 5 minutes overwhelming on-call phone, each alert identical except timestamp |
| Platform | Monitoring | ⚠️ Alerts lack context about scope, recent changes, or problem patterns; on-call engineer must gather basic information manually |
| Platform | Monitoring | ⚠️ All alerts route through same channel regardless of severity; critical production outages compete with minor warnings |
| Platform | Monitoring | ⚠️ Alerts fire during scheduled maintenance windows generating false positives that erode trust in monitoring system |
| Platform | Monitoring | ⚠️ Alert grouping does not exist; each transaction failure fires separate alert creating alert storms during incidents |
| Platform | Monitoring | ⚠️ Alerts provide no links to relevant dashboards; responders spend time finding monitoring tools during incident response |
On-Call Rotation and Runbook
Patterns
| Where to look | What good looks like |
|---|---|
| Platform | Operations | ✅ On-call rotation is weekly or bi-weekly balancing burden distribution against context preservation |
| Platform | Operations | ✅ Runbooks are documented for every alert type enabling on-call engineers to respond effectively without deep domain expertise |
| Platform | Operations | ✅ Runbooks include problem description, diagnostic steps, remediation procedures, escalation criteria, and related documentation links |
| Platform | Operations | ✅ Runbooks are living documents in version control that evolve through postmortem action items |
| Platform | Operations | ✅ Handoff procedures include brief meetings during rotation transitions transferring context including active issues, recent changes, known risks, and upcoming maintenance |
| Platform | Operations | ✅ Written handoff notes supplement verbal handoff preventing information loss during rotation transitions |
| Platform | Operations | ✅ Page frequency, after-hours incident count, and false positive rate are measured to identify unsustainable on-call burden requiring architectural remediation |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Platform | Operations | ⚠️ Runbooks do not exist; on-call engineers rely on tribal knowledge and frequent escalations to subject matter experts |
| Platform | Operations | ⚠️ Runbooks are stored in personal documents rather than shared version control where team members can access and update them |
| Platform | Operations | ⚠️ On-call rotation transitions happen without handoff meetings; incoming engineer lacks context about active issues and recent changes |
| Platform | Operations | ⚠️ High on-call burden continues indefinitely through operational heroics rather than driving architectural improvements |
| Platform | Operations | ⚠️ On-call metrics (page frequency, after-hours incidents, false positives) are not tracked; burden patterns remain invisible |
| Platform | Operations | ⚠️ Runbooks are created once during initial deployment and never updated as system evolves or new failure modes emerge |
Escalation Path
Patterns
| Where to look | What good looks like |
|---|---|
| Platform | Operations | ✅ Technical escalation paths define which subject matter experts handle specific problem domains |
| Platform | Operations | ✅ Management escalation paths define which managers receive notification for incidents exceeding severity or duration thresholds |
| Platform | Operations | ✅ Vendor escalation paths document Salesforce support case creation procedures and severity criteria |
| Platform | Operations | ✅ Executive escalation paths define which executives receive notification for business-critical incidents |
| Platform | Operations | ✅ Escalation criteria are explicit and objective: severity levels, time thresholds, affected user counts, business impact assessments |
| Platform | Operations | ✅ Escalation paths are documented in runbooks and accessible during incidents without searching for contact information |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Platform | Operations | ⚠️ Escalation paths are not documented; on-call engineers waste time searching for appropriate contacts during incidents |
| Platform | Operations | ⚠️ Escalation criteria are vague ("escalate if it seems bad"); responders lack clear guidance about when to escalate |
| Platform | Operations | ⚠️ Management notification happens too late because severity thresholds are undefined; executives learn about incidents from users |
| Platform | Operations | ⚠️ Salesforce support escalation procedures are not documented; teams delay opening support cases losing valuable response time |
| Platform | Operations | ⚠️ Subject matter expert contacts are outdated; escalations reach wrong people or fail during rotation changes |
Post-Incident Review Process
Patterns
| Where to look | What good looks like |
|---|---|
| Platform | Operations | ✅ Blameless post-incident reviews are conducted within 5 business days after significant operational events |
| Platform | Operations | ✅ Blameless culture focuses on system improvements rather than individual blame, creating psychological safety for honest assessment |
| Platform | Operations | ✅ Timeline includes detailed sequence from initial conditions through first signal, detection, acknowledgment, investigation, response, recovery, and validation |
| Platform | Operations | ✅ Timeline timestamps each event enabling duration analysis of detection, response, and recovery phases |
| Platform | Operations | ✅ Root cause identifies technical failure or condition that directly caused incident, distinguished from contributing factors |
| Platform | Operations | ✅ Contributing factors analysis reveals organizational, process, or architectural conditions that enabled or amplified incident impact |
| Platform | Operations | ✅ Detection analysis evaluates how incident was detected and whether detection could have been faster |
| Platform | Operations | ✅ Response evaluation identifies what went well during response and what was slower or harder than necessary |
| Platform | Operations | ✅ Action items are specific, assigned, and timebound preventing recurrence or improving future response |
| Platform | Operations | ✅ Action items avoid vague improvements ("improve monitoring") in favor of concrete tasks ("add alert for API error rate exceeding 5% over 10min window") |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Platform | Operations | ⚠️ Post-incident discussion concludes "batch job had a bug, we fixed it" with no documentation or systematic improvements |
| Platform | Operations | ⚠️ Post-incident reviews focus on individual fault rather than system improvements, destroying psychological safety |
| Platform | Operations | ⚠️ Timeline lacks timestamps preventing duration analysis; team cannot identify whether detection or response was slow |
| Platform | Operations | ⚠️ Root cause analysis does not distinguish immediate technical cause from organizational or process contributing factors |
| Platform | Operations | ⚠️ Detection analysis is skipped; monitoring gaps that allowed users to report incident first are not identified |
| Platform | Operations | ⚠️ Action items are vague without owners or due dates; improvements are discussed but never implemented |
| Platform | Operations | ⚠️ Post-incident reviews are not conducted within 5 business days; context is lost and urgency for improvements fades |
Post-Incident Documentation and Learning
Patterns
| Where to look | What good looks like |
|---|---|
| Platform | Knowledge Management | ✅ Post-incident reviews are documented and shared broadly beyond immediate incident responders for organizational learning |
| Platform | Knowledge Management | ✅ Post-incident reviews are stored in shared wiki or knowledge base with tagging enabling pattern analysis across multiple incidents |
| Platform | Knowledge Management | ✅ Action item completion is tracked in subsequent reviews preventing retrospectives from becoming repetitive discussion without follow-through |
| Platform | Knowledge Management | ✅ Action item completion rates are measured as operational health indicator revealing whether improvements are implemented or merely discussed |
| Platform | Operations | ✅ Incident patterns are analyzed across multiple incidents revealing systemic issues requiring architectural investment |
| Platform | Operations | ✅ Runbooks are updated based on post-incident learnings ensuring procedures reflect actual incident response experience |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Platform | Knowledge Management | ⚠️ Post-incident reviews are not documented; learnings remain with incident responders rather than spreading across organization |
| Platform | Knowledge Management | ⚠️ Post-incident reviews are stored in personal documents rather than shared knowledge base where team members can learn from history |
| Platform | Knowledge Management | ⚠️ Action items are defined but never tracked; completion rate is unknown and improvements are not implemented |
| Platform | Knowledge Management | ⚠️ Low action item completion rates persist without recognition that retrospectives are ineffective without follow-through |
| Platform | Operations | ⚠️ Incident patterns are not analyzed across multiple incidents; systemic issues remain hidden because each incident is treated in isolation |
| Platform | Operations | ⚠️ Runbooks are not updated after incidents; procedures become outdated as system evolves and new failure modes emerge |
Incident Metrics and Monitoring
Patterns
| Where to look | What good looks like |
|---|---|
| Platform | Operations | ✅ Mean time to detect (MTTD) is tracked measuring duration from incident occurrence to detection |
| Platform | Operations | ✅ Mean time to acknowledge (MTTA) is tracked measuring duration from detection to on-call engineer acknowledgment |
| Platform | Operations | ✅ Mean time to restore (MTTR) is tracked measuring duration from detection to service restoration |
| Platform | Operations | ✅ Incident frequency is tracked with monthly trend review revealing whether stability is improving or degrading |
| Platform | Operations | ✅ Change failure rate is tracked measuring percentage of deployments causing production incidents requiring remediation |
| Platform | Operations | ✅ Change failure rate is kept consistently low, a small fraction of all deployments |
| Platform | Operations | ✅ Incident metrics are shared with development teams during sprint planning informing backlog prioritization and architectural improvements |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Platform | Operations | ⚠️ Incident metrics are not tracked; team has no visibility into whether response times are improving or degrading |
| Platform | Operations | ⚠️ MTTD, MTTA, and MTTR are not measured; teams cannot identify whether detection or response is the bottleneck |
| Platform | Operations | ⚠️ Change failure rate is not tracked; team does not know whether quality gates are effective |
| Platform | Operations | ⚠️ Incident frequency trends are not reviewed monthly; stability degradation is not detected until major incident occurs |
| Platform | Operations | ⚠️ Incident metrics remain in operations team without sharing to development teams; architectural problems are not addressed |
| Platform | Operations | ⚠️ Metrics are tracked but never used to inform decisions; measurement without action wastes effort and provides no value |