Salesforce Docs Portal

Create Custom Scorers

Custom scorers evaluate AI agent sessions against specific criteria that you define, like tone, brand voice, or business outcomes. Use them alongside default scorers to get a complete picture of agent performance.

Required Editions

green checkmark

This article applies to:

 New Testing Center in Agentforce Studio (Beta)
red crossmark

This article doesn’t apply to:

 Legacy Agentforce Testing Center in Setup
Available in: Lightning Experience
Available in: Enterprise, Performance, Unlimited, and Developer Editions. Required add-on licenses vary by agent type.
User Permissions Needed
To create tests in the Testing Center:

Manage Agentforce Grids

AND

Manage Agentforce Testing

Along with default scorers, custom scorers let you define specific criteria to assess your agent's effectiveness. With custom scorers you can evaluate resolution quality, policy adherence, sentiment, and any business-specific outcome your team cares about. Or, use custom scorers when you expect a structured output like a JSON.

Custom scorers use an LLM-as-judge: you define the evaluation criteria in a prompt template, and the AI model uses it to assess each session and return a result based on the output type you configure.

Custom scorers return results in two ways based on the output type you select.

  • Labels. Classifies each session using a predefined set of values. Use this pattern for classification where outputs must stay within approved values.
  • Free-form. Returns output based on your prompt, like text, a number, or a URL. Use this pattern for richer insights like summaries, or sentiment signals.

Once you've decided which output type fits your use case, create a custom scorer in your test suite.

  1. To add or review your scorers from the test suite, click Select Scorers.
  2. Click Add and select LLM Judge.
    What is an LLM judge?

    An LLM judge (or LLM-as-judge) is when one large language model (LLM) evaluates and scores the outputs of another. A judge LLM receives a prompt that defines the task and outlines the scoring criteria like factual accuracy, relevance, coherence, and faithfulness to source. With these resources and guidelines, the LLM judge determines the expected response and compares it to the agent response. Based on this comparison, the judge generates scores, rankings, or written feedback. The LLM as judge model serves as a scalable, automated, and objective evaluation tool for tasks like scoring summaries or ranking responses. We’ve carefully designed our LLM-as-judge prompts to give you the most accurate and useful test results.

  3. Give the custom scorer a Name and API Name, and select a Response Data Type.
    • Select Labels to classify each session with one value from a predefined set you define. Labels constrain the AI model to return only values from your predefined set. You reference these values in your prompt template via merge fields, and Agentforce injects them at runtime so the AI model knows its valid options when evaluating each session.
    • Select Boolean, DateTime, Number, Text, or URL to return free-form output based on your prompt.
  4. If you selected Labels, configure the label fields.
    1. In Allowed Labels, enter the label values the AI model may apply (for example, TRUE, FALSE).

      These values are injected into your prompt via merge fields. Your prompt instructs the AI model to return exactly one of them as the scorer result for each session.

    2. In Fallback Label, enter the label to apply when the AI model can't determine a result or when prompt execution fails (for example, UNSURE).
  5. Create the evaluation prompt in the Prompt Template Composer.
    1. Select the AI model to serve as the judge.
    2. Write the evaluation prompt. For Labels scorers, include merge fields to supply the conversation transcript and your label values to the AI model at runtime.

      Your prompt should define the evaluation criteria and describe when each label applies. Merge fields supply runtime data the AI model can't access otherwise.

      Important
      Important Session-level and turn-level tests use different data models. Make sure the merge fields in your prompt match the test scope you plan to activate the scorer for.
      {!$Input:AllowedLabels}
      Tells the AI model which label values it can return. Without this, the model may generate values outside your defined set.
      {!$Input:FallbackLabel}
      Tells the AI model what to return when it can't determine a result.
      {!$SalesforceDataAction:getSession.chatTranscript}
      Supplies the full conversation transcript. Use for session-level (conversational) scoring.
      {!$SalesforceDataAction:getLastInteraction.userUtterance}
      Supplies the user's last message. Use for turn-level (intent) scoring.
      {!$SalesforceDataAction:getLastInteraction.agentResponse}
      Supplies the agent's last response. Use for turn-level (intent) scoring.

      For complete API reference, see Custom Scorers API Reference.

    3. Optionally, save multiple template versions.
    4. Set the threshold score for your passing criteria.
  6. Save your scorer.

After you save your custom scorer, it's automatically selected for your test suite. Once Agentforce finishes generating tests, they appear under Test Cases. Depending on the size and complexity of the request, this process can take anywhere from a few minutes to several hours. You may need to refresh the testing suite to see your test cases.

When the test cases appear, click Run Test. Agentforce begins running the chosen scorers on the test cases.