Build an Agent Testing and Observability Plan

Step 1: Establish your testing pathways in Agentforce Builder

The outcome area you already defined tells you which flows to run. If you've defined success as customers resolving billing questions without a human, run billing scenarios, including the hard cases. A testing scenario is a structured example used to evaluate how well an AI agent performs in a realistic situation. It simulates a real-world customer interaction and helps you test whether the agent understands the user input correctly, routes to the right subagent, runs appropriate actions, and responds accurately, completely, and in the right tone.

A strong testing scenario should include:

Scenario component	What to include
Scenario description	A short narrative of the situation or customer context, such as a customer who just received their device and is confused by an error code.
Test utterance	The actual phrase or question the customer might enter in natural language.
Expected subagent	The subagent the agent should classify the utterance into based on how the agent is configured, or the subagent the agent should route the conversation to based on deterministic transitions.
Expected action	The specific action the agent should take in response, like “lookup information about Makana's products and policies” or “escalate the conversation to a service rep.”
Expected response	A brand-appropriate reply based on validated information. The agent’s wording may vary, but as long as the tone feels right and the info is accurate, that’s what really counts.
Utterance variations	The different ways that customers are likely to phrase a question or request, including utterances that vary based on language differences, different emotional states of the customer, and utterances that change the topic of the conversation.
Multi-turn	It’s important to test how the agent handles requests inside of continued conversation.

To help you come up with good test scenarios, review the subagents assigned to your agent. Look at the classification description and fields that describe the capabilities and parameters your agent should operate within and each instruction that directs how the agent performs. Write or generate inputs that test against these details to help make sure your agent acts reliably in each scenario.

You may even want to create scenarios that cover several categories of conversation including:

Category	What to include
Happy path	User gives complete inputs; agent selects the correct subagent and agent action; action succeeds; response is concise and correct.
Clarification path	Missing ID, ambiguous intent, multiple matching records, unknown language, vague ask.
Out-of-scope	Competitor question, unsupported product, request for prohibited advice, no relevant source found.
Error path	Flow failure, API timeout, permission denied, no records found, duplicate record, retriever no-match.
Handoff path	User asks for a human, repeated frustration, high-risk transaction, failed resolution after two attempts.
Security and adversarial	Prompt injection, "ignore previous instructions," hidden data request, policy bypass, malicious document content.
Grounding	Expected answer exists; no answer exists; outdated answer; conflicting sources; old product name or jargon.
Latency	P50/P95 per turn, agent action, and retrieval; slow API; heavy context; repeated agent action chain.
Channel and language	Regional input, voice numeric capture, Slack short replies, WhatsApp template rules, email thread context.
Regression	Every production failure becomes a test case before the next release.

It's unlikely you'll need to test every category. Narrow down your categories to match how the agent is actually deployed. An internal employee agent that never talks to customers does not need the same adversarial or multilingual testing as a customer-facing service agent. For each test, use the Interaction Summary and other tools to watch which subagent is selected, which agent action runs, and where the agent breaks. When a session fails or gives a faulty answer, fix it and run it again until it passes.

When the scenarios you created are working, take them into Testing Center. Run only the scenarios you already got working, and only in the categories that apply. That is how you catch regressions a single chat will miss. You now have your test scenarios: the conversations that matter, working, with expected behavior written down.

Step 2: Translate your KPIs into scorers in Testing Center

In Testing Center, you'll test your agent at scale and match scorers to the KPIs you already chose. Scorers are how you measure those KPIs before deployment, and some of them measure the same thing again after go-live. They can also give you a quantitative threshold for whether the agent or agent version is ready to ship.

Generate cases from the subagents in your scenario list and use the test description to keep generation on the flows you planned.

For each of your identified KPIs, ask whether a Testing Center scorer measures it on a test case. If yes, turn that scorer on and put a threshold on it.

KPI	Scorer	What the score actually is
Quality Score	Completeness, Coherence, Conciseness	Quality of the answer on that utterance
Instruction Adherence	Instruction Adherence	Whether the reply followed the subagent instructions
Latency	Latency	How long that case took
Brand voice, compliance, or another rule you wrote	Custom scorer, created before the first run	Whatever you defined in the scorer prompt. After go-live, you can run a custom scorer on live sessions to watch the same criterion.
The agent picks the right work	Subagent Assertion, Action Assertion	Pass or fail on the expected subagent and agent action. Not a deflection rate.

Three scorers come with every suite: Subagent Assertion, Action Assertion, and Response Evaluation. Those cover routing accuracy, action accuracy, and response correctness, so you don't select them. Metrics like deflection and CSAT aren't available because they depend on live customer data. You can watch them after go-live. Choose the rest based on the KPIs you already have.

If built-in scorers don't measure one of your KPIs, consider defining a custom scorer. Build one when the KPI matters enough to your outcome area to track from pre-deployment through go-live, or if you have a specific ruleset or criteria to check against. Brand voice, tone, and compliance are all candidates for custom scorers.

You can use initial scores to set a pre-deployment baseline for your agent's performance. They can also be used as a gate for your ship decisions. For each scorer you enabled, write a floor before you run: how many cases must pass, or how high the score must be. The KPI decides which scorer to use, but the scorer's output is what you hold the agent to.

Set each floor from what the scorer's scale already means. Assertions are pass or fail, so primary scenarios get no failures. A quality score of 3 means the agent met the goal, so 3.5 is a reasonable bar. Latency uses the limit that channel already owes users. Raise the bar when the flow is high risk, and write the number before the run, because a floor set afterward just describes the results.

Once the run finishes, you'll see pass or fail on the assertions and a score for each quality scorer. Read those against the floors you wrote. Recurring failures are the useful ones: they usually point to unclear subagent instructions, missing knowledge, or a configuration problem rather than one bad reply.

You now have a test plan: those scenarios, the scorers that measure your KPIs, and the scores the agent has to hit to ship.

Step 3: Turn production KPIs into actions

After go-live, Agent Analytics gives you the KPIs the test suite couldn't: deflection, CSAT, error rate. It also gives you quality and latency again, this time on real conversations instead of test cases. Your custom criteria can run here too, so things like brand voice and compliance stay measured after launch. But before your session data pours in, develop a plan for reading it. That way you'll know what to pay attention to.

For each of your KPIs, find its source in Analytics. Some are standard and others need you to set them up. For example you'll need to set up your custom criteria as session scorers for them to carry over into analytics.

KPI	Where it comes from	What it tells you
Deflection rate	Analytics, Effectiveness. Standard Deflection Score runs on sessions.	Share of sessions resolved without handing off to a human
Abandonment rate	Analytics. Standard Abandon Score.	Customers who left before the issue was resolved
Escalation rate	Analytics, Effectiveness	How often the agent hands off
Quality Score	Analytics, Quality Scores. Standard Response Quality Score.	The same quality signal you scored in testing, now on real conversations
Latency	Analytics, Health. Session Trace for per-step timing.	Where real sessions are slow, not just that they are
Error rate	Analytics, Health	System and agent failures in production
Instruction adherence	Instruction adherence monitor, guardrails events	Whether live responses stayed inside your instructions
CSAT, NPS, thumbs	Feedback signals, Analytics	What customers say about the interaction
Brand voice, compliance, your own rules	Custom scorer on live sessions	Whatever you defined in the scorer prompt
Resolution, conversion, adoption	Analytics business value dashboards, plus CRM data	Whether the outcome area moved

For each KPI you'll want to set a target for where you expect the agent to land after several weeks of real traffic. Write a target, how far it can drift before you investigate, a date to revisit it, and the work flow it applies to. "Quality Score at or above 4.0 and no lower than 3.5 on billing questions." is something you can check. Set up alerts with the target threshold so that you’re automatically notified when agent performance falls above or below the threshold.

To set realistic goals, first look at the data you already have. For quality and latency, the suite results are your baseline. You just measured those on test cases, so you know roughly where the agent performs. A production target near or slightly above that is defensible. This is the one place where looking at scores first is fine, because you're setting a future expectation, not grading the run you already finished.

For deflection, CSAT, and resolution, the target comes from the business case. If the goal was cutting billing contacts by a third, that's your number. If people handle this work today, their current rate is your reference point.

Custom scorers usually return a label, so the target is a rate. For a compliance rule, that rate is the rule itself — if the agent must never give dosage advice, the target is zero failures. For brand voice or tone, start from what the scorer returned on your suite and allow for real conversations being messier. Remember that live scorers run on a sample of sessions, so the rate reflects that sample. Either way you're estimating, which is why each target gets a revisit date.

You should look at each KPI as often as it produces something you can act on. Error rate and latency move by the hour, so a daily check catches an outage while it's happening. Quality needs a batch of sessions before the average means anything. Things like Deflection and CSAT need even more still.

KPI type	How often	Why	Owner	Example: billing agent
Impact: deflection, CSAT, resolution	Monthly	Needs the most volume before the number means anything.	Support lead and ops	Deflection more than 10 points below target
Quality: Quality Score, Instruction Adherence, custom scorers	Weekly	Needs volume before an average means anything.	Support lead	Quality below 3.5 on billing questions, or brand voice pass rate below 95%
Health: error rate, latency, availability	Daily	A spike can be an outage, not a trend.	Platform admin	Latency above 2 seconds at P95, or any error spike

Having a clear plan around cadence helps separate noise from signals. One session with a low score is noise. A downward trend over several weeks is a signal.

You've set what to watch, how often, and what counts as a problem. What's missing is what happens next, because knowing a number moved isn't the same as knowing what to do about it. For every scorer you’re using, identify where it surfaces, and the follow up action to take when it moves.

Signal	Where to look	What to do
Deflection drops	Effectiveness, then Agent Optimization for escalated sessions, then Trace	Roll back a failing agent action and retest, route out-of-scope work to a human, or change the target if quality is fine
Quality Score drops	Quality Scores, then the failing sessions, then Trace	Roll back the instruction or prompt template change, fix the knowledge source, or isolate one bad intent cluster
Latency rises	Health, then Trace for per-step timing	Fix the slow step. If quality dropped too, treat it as a quality problem rather than a speed win
Escalation spikes	Effectiveness, then Agent Optimization	Check abandonment first. If people are leaving, this isn't healthy deflection
A KPI holds above target	The flows driving it	Name that flow as stable, expand to an adjacent flow, or check it less often

A KPI moving the wrong way doesn't always mean the agent is broken. When Agent Optimization surfaces a session that underperformed, that session belongs in your suite. Fix the cause, retest in Agentforce Testing Center, then watch the KPI recover. If it doesn't move, the diagnosis was wrong, so go back to the sessions rather than shipping another change on top.

You now have a monitoring plan: what to watch, how often, and what each signal triggers.