--- schema_version: "1" id: 9db62908add28b69a0729ec24d8ec2159e2b96bed4aa56c1ffef8f75cd56e499 title: Incident Management - Patterns collection: architect version: next locale: en-us portal_url: https://docs.salesforce.com/architect/next/en-us/well-architected/guides/operational-excellence-incident-patterns.md canonical_url: https://staging.architect.salesforce.com/docs/architect/well-architected/guide/operational-excellence-incident-patterns.html source_site: staging.architect.salesforce.com content_type: documentation content_hash: 161e27a543a3c19090df400c893f9f1c609f967c22c971ec0860a9c2b63e3b12 products: - Well-Architected Framework - Architect Site Content Repo --- # Incident Management - Patterns Learn more about Well-Architected [Operational Excellence](/docs/architect/well-architected/guide/operational-excellence.html) → [Incident Management](/docs/architect/well-architected/guide/operational-excellence.html#incident-management) ## Incident Detection and Alerting **Patterns** | Where to look | What good looks like | | --- | --- | | Platform \| Monitoring | ✅ Alerting architecture includes severity classification routing critical incidents (production unavailable, data corruption, security compromise) to page on-call immediately | | Platform \| Monitoring | ✅ High severity incidents (degraded performance, partial outage) generate immediate notifications for prompt business-hours investigation | | Platform \| Monitoring | ✅ Warnings (approaching thresholds, configuration drift) aggregate into daily digest for proactive remediation | | Platform \| Monitoring | ✅ Alert enrichment includes diagnostic context: affected user count, error rate percentiles, recent deployments, related alerts, and direct dashboard links | | Platform \| Monitoring | ✅ Alert context reduces mean time to acknowledge by providing immediate situation assessment without gathering basic information through separate queries | | Platform \| Monitoring | ✅ Alert grouping prevents alert storms by consolidating hundreds of related alerts from single underlying problem into one notification with aggregate statistics | | Platform \| Monitoring | ✅ Grouped alerts show count, rate, and affected transactions revealing problem pattern without overwhelming responders | | Platform \| Monitoring | ✅ Alert suppression prevents known-transient conditions from generating alerts during expected maintenance windows or auto-remediation periods | **Anti-Patterns** | Where to look | What bad looks like | | --- | --- | | Platform \| Monitoring | ⚠️ Integration failure generates 100s individual alerts in 5 minutes overwhelming on-call phone, each alert identical except timestamp | | Platform \| Monitoring | ⚠️ Alerts lack context about scope, recent changes, or problem patterns; on-call engineer must gather basic information manually | | Platform \| Monitoring | ⚠️ All alerts route through same channel regardless of severity; critical production outages compete with minor warnings | | Platform \| Monitoring | ⚠️ Alerts fire during scheduled maintenance windows generating false positives that erode trust in monitoring system | | Platform \| Monitoring | ⚠️ Alert grouping does not exist; each transaction failure fires separate alert creating alert storms during incidents | | Platform \| Monitoring | ⚠️ Alerts provide no links to relevant dashboards; responders spend time finding monitoring tools during incident response | ## On-Call Rotation and Runbook **Patterns** | Where to look | What good looks like | | --- | --- | | Platform \| Operations | ✅ On-call rotation is weekly or bi-weekly balancing burden distribution against context preservation | | Platform \| Operations | ✅ Runbooks are documented for every alert type enabling on-call engineers to respond effectively without deep domain expertise | | Platform \| Operations | ✅ Runbooks include problem description, diagnostic steps, remediation procedures, escalation criteria, and related documentation links | | Platform \| Operations | ✅ Runbooks are living documents in version control that evolve through postmortem action items | | Platform \| Operations | ✅ Handoff procedures include brief meetings during rotation transitions transferring context including active issues, recent changes, known risks, and upcoming maintenance | | Platform \| Operations | ✅ Written handoff notes supplement verbal handoff preventing information loss during rotation transitions | | Platform \| Operations | ✅ Page frequency, after-hours incident count, and false positive rate are measured to identify unsustainable on-call burden requiring architectural remediation | **Anti-Patterns** | Where to look | What bad looks like | | --- | --- | | Platform \| Operations | ⚠️ Runbooks do not exist; on-call engineers rely on tribal knowledge and frequent escalations to subject matter experts | | Platform \| Operations | ⚠️ Runbooks are stored in personal documents rather than shared version control where team members can access and update them | | Platform \| Operations | ⚠️ On-call rotation transitions happen without handoff meetings; incoming engineer lacks context about active issues and recent changes | | Platform \| Operations | ⚠️ High on-call burden continues indefinitely through operational heroics rather than driving architectural improvements | | Platform \| Operations | ⚠️ On-call metrics (page frequency, after-hours incidents, false positives) are not tracked; burden patterns remain invisible | | Platform \| Operations | ⚠️ Runbooks are created once during initial deployment and never updated as system evolves or new failure modes emerge | ## Escalation Path **Patterns** | Where to look | What good looks like | | --- | --- | | Platform \| Operations | ✅ Technical escalation paths define which subject matter experts handle specific problem domains | | Platform \| Operations | ✅ Management escalation paths define which managers receive notification for incidents exceeding severity or duration thresholds | | Platform \| Operations | ✅ Vendor escalation paths document Salesforce support case creation procedures and severity criteria | | Platform \| Operations | ✅ Executive escalation paths define which executives receive notification for business-critical incidents | | Platform \| Operations | ✅ Escalation criteria are explicit and objective: severity levels, time thresholds, affected user counts, business impact assessments | | Platform \| Operations | ✅ Escalation paths are documented in runbooks and accessible during incidents without searching for contact information | **Anti-Patterns** | Where to look | What bad looks like | | --- | --- | | Platform \| Operations | ⚠️ Escalation paths are not documented; on-call engineers waste time searching for appropriate contacts during incidents | | Platform \| Operations | ⚠️ Escalation criteria are vague ("escalate if it seems bad"); responders lack clear guidance about when to escalate | | Platform \| Operations | ⚠️ Management notification happens too late because severity thresholds are undefined; executives learn about incidents from users | | Platform \| Operations | ⚠️ Salesforce support escalation procedures are not documented; teams delay opening support cases losing valuable response time | | Platform \| Operations | ⚠️ Subject matter expert contacts are outdated; escalations reach wrong people or fail during rotation changes | ## Post-Incident Review Process **Patterns** | Where to look | What good looks like | | --- | --- | | Platform \| Operations | ✅ Blameless post-incident reviews are conducted within 5 business days after significant operational events | | Platform \| Operations | ✅ Blameless culture focuses on system improvements rather than individual blame, creating psychological safety for honest assessment | | Platform \| Operations | ✅ Timeline includes detailed sequence from initial conditions through first signal, detection, acknowledgment, investigation, response, recovery, and validation | | Platform \| Operations | ✅ Timeline timestamps each event enabling duration analysis of detection, response, and recovery phases | | Platform \| Operations | ✅ Root cause identifies technical failure or condition that directly caused incident, distinguished from contributing factors | | Platform \| Operations | ✅ Contributing factors analysis reveals organizational, process, or architectural conditions that enabled or amplified incident impact | | Platform \| Operations | ✅ Detection analysis evaluates how incident was detected and whether detection could have been faster | | Platform \| Operations | ✅ Response evaluation identifies what went well during response and what was slower or harder than necessary | | Platform \| Operations | ✅ Action items are specific, assigned, and timebound preventing recurrence or improving future response | | Platform \| Operations | ✅ Action items avoid vague improvements ("improve monitoring") in favor of concrete tasks ("add alert for API error rate exceeding 5% over 10min window") | **Anti-Patterns** | Where to look | What bad looks like | | --- | --- | | Platform \| Operations | ⚠️ Post-incident discussion concludes "batch job had a bug, we fixed it" with no documentation or systematic improvements | | Platform \| Operations | ⚠️ Post-incident reviews focus on individual fault rather than system improvements, destroying psychological safety | | Platform \| Operations | ⚠️ Timeline lacks timestamps preventing duration analysis; team cannot identify whether detection or response was slow | | Platform \| Operations | ⚠️ Root cause analysis does not distinguish immediate technical cause from organizational or process contributing factors | | Platform \| Operations | ⚠️ Detection analysis is skipped; monitoring gaps that allowed users to report incident first are not identified | | Platform \| Operations | ⚠️ Action items are vague without owners or due dates; improvements are discussed but never implemented | | Platform \| Operations | ⚠️ Post-incident reviews are not conducted within 5 business days; context is lost and urgency for improvements fades | ## Post-Incident Documentation and Learning **Patterns** | Where to look | What good looks like | | --- | --- | | Platform \| Knowledge Management | ✅ Post-incident reviews are documented and shared broadly beyond immediate incident responders for organizational learning | | Platform \| Knowledge Management | ✅ Post-incident reviews are stored in shared wiki or knowledge base with tagging enabling pattern analysis across multiple incidents | | Platform \| Knowledge Management | ✅ Action item completion is tracked in subsequent reviews preventing retrospectives from becoming repetitive discussion without follow-through | | Platform \| Knowledge Management | ✅ Action item completion rates are measured as operational health indicator revealing whether improvements are implemented or merely discussed | | Platform \| Operations | ✅ Incident patterns are analyzed across multiple incidents revealing systemic issues requiring architectural investment | | Platform \| Operations | ✅ Runbooks are updated based on post-incident learnings ensuring procedures reflect actual incident response experience | **Anti-Patterns** | Where to look | What bad looks like | | --- | --- | | Platform \| Knowledge Management | ⚠️ Post-incident reviews are not documented; learnings remain with incident responders rather than spreading across organization | | Platform \| Knowledge Management | ⚠️ Post-incident reviews are stored in personal documents rather than shared knowledge base where team members can learn from history | | Platform \| Knowledge Management | ⚠️ Action items are defined but never tracked; completion rate is unknown and improvements are not implemented | | Platform \| Knowledge Management | ⚠️ Low action item completion rates persist without recognition that retrospectives are ineffective without follow-through | | Platform \| Operations | ⚠️ Incident patterns are not analyzed across multiple incidents; systemic issues remain hidden because each incident is treated in isolation | | Platform \| Operations | ⚠️ Runbooks are not updated after incidents; procedures become outdated as system evolves and new failure modes emerge | ## Incident Metrics and Monitoring **Patterns** | Where to look | What good looks like | | --- | --- | | Platform \| Operations | ✅ Mean time to detect (MTTD) is tracked measuring duration from incident occurrence to detection | | Platform \| Operations | ✅ Mean time to acknowledge (MTTA) is tracked measuring duration from detection to on-call engineer acknowledgment | | Platform \| Operations | ✅ Mean time to restore (MTTR) is tracked measuring duration from detection to service restoration | | Platform \| Operations | ✅ Incident frequency is tracked with monthly trend review revealing whether stability is improving or degrading | | Platform \| Operations | ✅ Change failure rate is tracked measuring percentage of deployments causing production incidents requiring remediation | | Platform \| Operations | ✅ Change failure rate is kept consistently low, a small fraction of all deployments | | Platform \| Operations | ✅ Incident metrics are shared with development teams during sprint planning informing backlog prioritization and architectural improvements | **Anti-Patterns** | Where to look | What bad looks like | | --- | --- | | Platform \| Operations | ⚠️ Incident metrics are not tracked; team has no visibility into whether response times are improving or degrading | | Platform \| Operations | ⚠️ MTTD, MTTA, and MTTR are not measured; teams cannot identify whether detection or response is the bottleneck | | Platform \| Operations | ⚠️ Change failure rate is not tracked; team does not know whether quality gates are effective | | Platform \| Operations | ⚠️ Incident frequency trends are not reviewed monthly; stability degradation is not detected until major incident occurs | | Platform \| Operations | ⚠️ Incident metrics remain in operations team without sharing to development teams; architectural problems are not addressed | | Platform \| Operations | ⚠️ Metrics are tracked but never used to inform decisions; measurement without action wastes effort and provides no value |