Learn more about Well-Architected Reliability → Agentic Enterprise Reliability
Agent Failure Mode Detection
Patterns
| Where to look | What good looks like |
|---|---|
| Platform | Platform Cache | ✅ Implement circuit breakers that track LLM service health state in Platform Cache for low-latency real-time reads, backed by Custom Metadata Types for thresholds and configuration; because Platform Cache is best-effort, persist the open/tripped state in a durable store (such as a Custom Object) rather than the cache alone. Open the circuit on a tuned failure threshold (for example, five consecutive inference failures) and route to fallback patterns |
| Platform | Platform Cache | ✅ Store validated response schemas in Platform Cache enabling fast validation of agent responses against expected structure |
| Agentforce | Prompt Template | ✅ When response validation fails, retry with refined prompts including the validation error and an example of correct structure (maximum 3-5 retries) |
| Agentforce | Session Tracing | ✅ Use Agentforce Session Tracing reasoning traces to inspect the agent's reasoning path and detect hallucination in its outputs |
| Data 360 | Vector Search | ✅ Require agents to cite Data 360 vector search sources so claims can be cross-referenced against retrieved sources before action execution |
| Platform | Proactive Monitoring | ✅ For accounts with the Signature Success entitlement, rely on Proactive Monitoring, where Salesforce staff monitor the org for emerging issues; it is not a self-service configurable alert |
| Platform | Scale Center | ✅ Use Scale Center to identify specific agent operations approaching governor limits |
| Agentforce | Action | ✅ Implement pagination in agent actions retrieving large record collections |
| Agentforce | Action | ✅ Design agent actions with hard limits (maximum 200 children per parent) and implement sampling strategies when limits are exceeded |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Platform | Custom Metadata Types | ⚠️ Allowing unlimited inference retries without a circuit breaker, creating infinite retry loops when LLM service is degraded |
| Agentforce | Prompt Template | ⚠️ Retrying failed agent responses without including the validation error and expected structure in the retry prompt |
| Agentforce | Action | ⚠️ Allowing agents to request unbounded context (e.g., "all history") without hard limits on children per parent record |
| Agentforce | Action | ⚠️ Designing agent actions that retrieve large record collections without pagination, causing governor limit spikes |
| Einstein | Trust Layer | ⚠️ Trusting agent responses without cross-referencing claims against grounding data sources, allowing hallucinations to reach action execution |
| Platform | Org | ⚠️ Treating LLM service availability as identical to Salesforce platform availability instead of monitoring it as a separate dependency |
Graceful Degradation
Patterns
| Where to look | What good looks like |
|---|---|
| Agentforce | Agent Configuration | ✅ Design multi-tier fallback chains: full-context agent → reduced-context agent → deterministic rule engine → human handoff via Omni-Channel |
| Agentforce | Agent Configuration | ✅ Configure a secondary agent using condensed context (last 30 days vs all history, top 5 vector search results vs top 20) to reduce failure probability |
| Platform | Flow | ✅ Implement fallback logic in orchestration Flow or Apex triggered by Platform Events, logging which tier succeeded for visibility into failure patterns |
| Platform | Flow | ✅ Build deterministic Flow or Apex logic handling common scenarios as fallback when agent reasoning fails (e.g., standard validation rules, rule-based score calculation) |
| Agentforce | Omni-Channel | ✅ Configure skill-based routing for human handoff ensuring escalations reach appropriate expertise |
| Agentforce | Omni-Channel | ✅ Pass full conversation context to human queue including attempted strategies, agent confidence scores, and failure reasons |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Agentforce | Agent Configuration | ⚠️ Designing agents with a single execution path that fails completely when any component degrades |
| Agentforce | Agent Configuration | ⚠️ Using the same full-context agent configuration for fallback attempts instead of progressively simplifying context |
| Agentforce | Omni-Channel | ⚠️ Routing to human handoff without passing conversation context, attempted strategies, or failure reasons, forcing the agent to start from scratch |
| Platform | Flow | ⚠️ Not logging which fallback tier succeeded, losing visibility into failure patterns and fallback effectiveness |
Platform Event Resilience
Patterns
| Where to look | What good looks like |
|---|---|
| Platform | Platform Events | ✅ Publish checkpoint events after each successful step in multi-step agent workflows, storing correlation ID, completed steps, and state needed for continuation in the event payload |
| Platform | Platform Events | ✅ Design agent workflows as loosely coupled event chains where each agent can fail and retry independently without blocking the entire workflow |
| Platform | Platform Events | ✅ Implement retry queues with exponential backoff delays (30s, 2min, 8min, 30min) using Platform Events with retry count in payload |
| Platform | Platform Events | ✅ Route to dead letter queue and alert operations after maximum retries (typically 5) are exhausted |
| Platform | Platform Events | ✅ Include correlation ID in all related events; use Platform Event Replay API to reconstruct complete workflow history for debugging |
| Platform | Platform Events | ✅ Leverage the 72-hour replay retention window available to high-volume platform events (standard-volume events retain for 24 hours) to preserve retry queues through org maintenance windows or deployment downtime |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Platform | Platform Events | ⚠️ Building multi-step agent workflows without checkpointing, requiring full re-execution when any intermediate step fails |
| Platform | Platform Events | ⚠️ Retrying failed agent requests without exponential backoff, creating thundering herd effects on recovering services |
| Platform | Platform Events | ⚠️ Designing tightly coupled sequential workflows where a single agent failure blocks all downstream processing |
| Platform | Platform Events | ⚠️ Omitting correlation IDs from events, making it impossible to trace and debug complete workflow execution |
| Platform | Platform Events | ⚠️ Retrying indefinitely without a maximum retry count or dead letter queue, consuming platform resources on permanently failed operations |
Async Processing
Patterns
| Where to look | What good looks like |
|---|---|
| Platform | Apex | ✅ Use Batch Apex for agent operations analyzing 1,000+ records, leveraging async governor limits (200 SOQL queries, 150 DML operations, 60s CPU per execute) |
| Platform | Apex | ✅ Use Queueable Apex chains for multi-step agent workflows requiring multiple external API calls or extended processing time; track progress in custom objects enabling resume on failure |
| Platform | Apex | ✅ Use @future methods for lightweight fire-and-forget agent operations (notifications, external logging, non-urgent updates) where failure is tolerable |
| Platform | Apex Jobs | ✅ Monitor async processing capacity via the Apex Jobs page (Setup → Apex Jobs) and the Apex Flex Queue: maximum 5 concurrent batch jobs (additional jobs queue in the Flex Queue, up to 100 in Holding status), and the org-wide daily limit on async Apex executions (DailyAsyncApexExecutions) — the greater of 250,000 or 200 × the number of user licenses per 24 hours — shared across Batch, Queueable, Future, and Scheduled Apex |
| Platform | Org | ✅ Alert when async consumption approaches limits enabling proactive capacity management before agent processing is blocked |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Platform | Apex | ⚠️ Processing 1,000+ records in synchronous agent transactions instead of moving to Batch Apex, hitting the 100 SOQL query limit after only a few records |
| Platform | Apex | ⚠️ Using Queueable chains without tracking progress in custom objects, preventing resume-on-failure capability |
| Platform | Org | ⚠️ Not monitoring the 5 concurrent batch job limit or the shared daily async Apex execution limit (DailyAsyncApexExecutions — the greater of 250,000 or 200 × user licenses per 24 hours, across Batch, Queueable, Future, and Scheduled Apex), causing agent processing to silently queue or fail |
Agent Reliability Testing
Patterns
| Where to look | What good looks like |
|---|---|
| Platform | Org | ✅ Load test agent performance in Full Copy Sandbox with production-scale data volumes; simulate concurrent conversations matching peak demand |
| Platform | Org | ✅ Measure p95 and p99 latency under load identifying performance degradation under stress; validate governor limit consumption stays below thresholds at peak load |
| Agentforce | Agent Configuration | ✅ Perform fault injection: simulate LLM inference failures in the service layer your circuit breaker monitors, inject SOQL exceptions testing error handling, corrupt responses testing validation logic |
| Agentforce | Agent Configuration | ✅ Introduce data skew (accounts with 10,000+ children) during testing to validate pagination logic handles worst-case scenarios |
| Platform | Platform Events | ✅ Conduct chaos engineering: randomly disable Platform Event subscribers testing replay recovery, terminate async jobs mid-execution testing checkpoint restart |
| Platform | Org | ✅ Schedule regular chaos days in staging environments maintaining resilience as agents evolve |
| Platform | Org | ✅ Execute long-running stability tests (72+ hours) monitoring for time-dependent failures: accumulating errors and gradually degrading performance |
| Platform | Org | ✅ Track CPU time trends and heap consumption over extended periods detecting gradual degradation in context management |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Platform | Org | ⚠️ Load testing in Partial Copy Sandbox with reduced data volumes, providing misleading confidence since data volume directly affects agent performance |
| Agentforce | Agent Configuration | ⚠️ Testing only happy paths without deliberately introducing failures to validate circuit breakers, retry logic, and timeout handling |
| Platform | Org | ⚠️ Running short-duration tests only, missing time-dependent failures (accumulating errors, gradual degradation) that appear only under sustained 72+ hour operation |
| Platform | Platform Events | ⚠️ Never testing Platform Event replay recovery or checkpoint restart, leaving resilience mechanisms unvalidated |
Agent SLI Monitoring
Patterns
| Where to look | What good looks like |
|---|---|
| Business | Documentation | ✅ Establish SLOs before building agents, not after production failures; define conversation success rate targets (typically 95-99%) per agent type |
| Platform | Org | ✅ Track conversation success rate as primary agent reliability metric: percentage of conversations completing without errors or timeouts. Count a graceful handoff to a human as a success, since escalation is the designed final fallback tier; count only failed or unintended escalations against the metric |
| Platform | Org | ✅ Track success rate separately by agent type and use case since success criteria differ between agent purposes |
| Platform | Org | ✅ Calculate Mean Time Between Failures (MTBF) per agent type to identify least reliable agents requiring optimization investment |
| Platform | Org | ✅ Measure Mean Time to Recovery (MTTR); automate recovery via Platform Event replay and circuit breakers for much faster, less error-prone recovery than manual intervention |
| Einstein | Trust Layer | ✅ Sample random agent responses validating claims against Data 360 sources to measure grounding accuracy; investigate when accuracy drops below 95% |
| Platform | Platform Events | ✅ Monitor Platform Event replay gaps (events missed or undelivered) via a subscriber-side logging pattern that records each event's replayId and correlation ID to a custom object |
| Platform | Org | ✅ Alert operations when any SLI drops below its defined SLO threshold |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Business | Documentation | ⚠️ Building agents without defining SLOs first, only establishing reliability targets after production failures occur |
| Platform | Org | ⚠️ Using a single aggregate success rate across all agent types instead of tracking separately by agent type and use case |
| Platform | Org | ⚠️ Relying solely on traditional Salesforce monitoring (trust.salesforce.com) without tracking agent-specific SLIs like conversation success rate, MTBF, MTTR, and grounding accuracy |
| Platform | Org | ⚠️ Requiring manual intervention for all agent failures instead of automating recovery with Platform Event replay and circuit breakers |
| Einstein | Trust Layer | ⚠️ Never sampling agent responses for grounding accuracy validation, allowing hallucination to go undetected in production |
Reliable Agent Design
Patterns
| Where to look | What good looks like |
|---|---|
| Agentforce | Agent Configuration | ✅ Design retry patterns, fallback chains, and escalation thresholds into agent workflows from initial design, not as afterthoughts |
| Agentforce | Action | ✅ Bulkify context preparation and paginate large record collections in agent actions |
| Platform | Platform Cache | ✅ Cache frequently accessed reference data in Platform Cache reducing redundant queries during agent context preparation |
| Platform | Scale Center | ✅ Use Scale Center to identify resource-intensive agent operations requiring optimization |
| Platform | Platform Events | ✅ Design self-recovering workflows using Platform Events as durable retry queues with circuit breakers protecting external dependencies |
| Agentforce | Agent Configuration | ✅ Always include human handoff as final fallback in every agent's degradation chain |
| Platform | Org | ✅ Monitor agent SLIs continuously alerting when metrics drop below SLOs |
| Agentforce | Agent Configuration | ✅ Balance autonomous capability with appropriate oversight: enable self-recovery from transient failures while escalating when confidence drops or edge cases emerge |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Agentforce | Agent Configuration | ⚠️ Adding reliability mechanisms (retry, fallback, escalation) as afterthoughts rather than designing them in from the start |
| Agentforce | Action | ⚠️ Not bulkifying context preparation, causing governor limit failures under real-world query patterns |
| Platform | Org | ⚠️ Not monitoring API consumption and Platform Event allocation proactively, discovering limits only when agents fail |
| Agentforce | Agent Configuration | ⚠️ Designing agent chains without a human handoff as final fallback, leaving no recovery path when all automated options are exhausted |
| Agentforce | Agent Configuration | ⚠️ Granting agents fully autonomous operation without escalation thresholds for low-confidence situations or edge cases |