---
schema_version: "1"
id: c65b5c347ba7a6be6743796f327eb690715c03d13ef281bc2f1dfbb7471602bd
title: Metadata API Reference for Testing API
collection: developer
version: current
locale: en-us
portal_url: https://docs.salesforce.com/developer/current/en-us/genai/agentforce/references/testing/testing-metadata-reference.md
canonical_url: https://developer.salesforce.com/docs/ai/agentforce/references/testing-api/testing-metadata-reference.html
source_site: developer.salesforce.com
content_type: documentation
content_hash: 019c9e31544ba9b41f364ff838fd9281f621d50aa5e6ef61f8888ff06030700d
products:
  - Agentforce Developer Guide
  - Agentforce and Einstein Generative AI
---
# Metadata API Reference for Testing API

To define tests, use the `AiEvaluationDefinition` Metadata API type.

## AiEvaluationDefinition

Represents an evaluation's purpose, the agent being tested, and the test sets to be executed.

In Metadata API, you can create test definitions, including specifying inputs and expected outcomes, and deploy them to different orgs. In Connect API, you can execute test scenarios, poll for results, and retrieve test outcomes.  
This type extends the [Metadata](https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/metadata.htm) metadata type and inherits its `fullName` field.

### File Suffix and Directory Location

AiEvaluationDefinition components have the suffix `.aiEvaluationDefinition` and are stored in the `aiEvaluationDefinitions` folder.

### Version

AiEvaluationDefinition components are available in API version 63.0 and later.

### Special Access Rules

AiEvaluationDefinition is available only if Agentforce is enabled. See [Set Up Your Agent](https://help.salesforce.com/s/articleView?id=ai.copilot_setup.htm) in Salesforce Help.

### Fields

| Field Name       | Type                     | Description                                                                                                                                                                                                                                            |
| :--------------- | :----------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `description`    | string                   | The purpose of the test.                                                                                                                                                                                                                               |
| `name`           | string                   | Required. The API name of the test. Can contain only underscores and alphanumeric characters and must be unique in your org. It must begin with a letter, not include spaces, not end with an underscore, and not contain two consecutive underscores. |
| `subjectName`    | string                   | Required. A unique identifier for the agent being tested. Make sure that this identifier matches the API name of the agent, which you can find on the agent details page in Setup.                                                                     |
| `subjectType`    | string                   | Required. The type of subject being tested. The only currently supported value is `AGENT`.                                                                                                                                                             |
| `subjectVersion` | string                   | The agent version to test. If not provided, the latest active version is used by default. You can find the version in the BotVersion metadata type.                                                                                                    |
| `testCase`       | AiEvaluationTestCase\[\] | A list of test cases.                                                                                                                                                                                                                                  |

#### AiEvaluationTestCase

Represents a test case.

| Field Name    | Type                               | Description                                                                                  |
| :------------ | :--------------------------------- | :------------------------------------------------------------------------------------------- |
| `expectation` | AiEvaluationExpectation\[\]        | The criteria used to test the artifact's responses.                                          |
| `inputs`      | AiEvaluationAgentTestCaseInput\[\] | The specific input provided to the artifact being tested.                                    |
| `number`      | int                                | The unique number for the test case. If not provided, the value is automatically calculated. |

#### AiEvaluationExpectation

Represents the expected outcome for a test case.

| Field Name      | Type                                             | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| :-------------- | :----------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `expectedValue` | string                                           | The expected outcome of the test. The format of this field depends on the value of the `name` field. The expected outcome is compared against the response generated when you run the test using [Connect REST API](/docs/ai/agentforce/references/testing-api/testing-connect-reference.html).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `label`         | string                                           | An optional label for an expectation. Typically added when using the same custom expectation name multiple times in a test case. If provided, this label appears in the test results; otherwise, the expectation name appears.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `name`          | string                                           | Required. The expectation name. Valid values are: <ul><li>`topic_sequence_match`&mdash;The `expectedValue` field value is a string representing the subagent that the agent is expected to use, such as `OOTBSingleRecordSummary`. For a list of subagents, see [Standard Subagent Reference](https://help.salesforce.com/s/articleView?id=ai.copilot_topics_ref.htm) in Salesforce Help.</li><li>`action_sequence_match`&mdash;The `expectedValue` field value is a string\[\] representing a list of actions that you expect the artifact to take during the test, such as `['IdentifyRecordByName', 'action2']`. For a list of agent actions, see [Standard Agent Action Reference](https://help.salesforce.com/s/articleView?id=ai.copilot_actions_ref.htm) in Salesforce Help. This option was previously called `action_sequence_match`.</li><li>`bot_response_rating`&mdash;The `expectedValue` field value is a string representing the expected response generated by the artifact, such as `Summarization of the Global Media account`.</li><li>`coherence`&mdash;A generated answer is coherent if it's easy to understand and has no grammatical errors. If you use this quality check, you don't need an `expectedValue` field value.</li><li>`completeness`&mdash;A generated answer is complete if it includes all the essential information. If you use this quality check, you don't need an `expectedValue` field value.</li><li>`conciseness`&mdash;A generated answer is concise if it's brief but comprehensive. Shorter is better. If you use this quality check, you don't need an `expectedValue` field value.</li><li>`output_latency_milliseconds`&mdash;Latency in milliseconds from sending a request until a response is received. If you use this quality check, you don't need an `expectedValue` field value.</li><li>`string_comparison`&mdash;A custom evaluation criteria that tests a response for a specified string value.</li><li>`numeric_comparison`&mdash;A custom evaluation criteria that tests a response for a specified numeric value.</li></ul> |
| `parameter`     | [parameter[]](#aievaluationexpectationparameter) | Required for custom test criteria. An array of parameters for the specific custom criteria defined by `expectation.name`. This field replaces `expectedValue` for custom test criteria.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |

#### AiEvaluationExpectationParameter

| Field Name    | Type    | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| :------------ | :------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`        | string  | Required for custom evaluation criteria. The name of the parameter required by the evaluation. Valid values are: <ul><li> `operator`&mdash;Specifies the type of string or numeric comparison.<ul><li>`equals`&mdash;Checks if the `actual` value directly `matches` the expected value.</li><li>`contains`&mdash;Checks if the `actual` value contains the `expected` value.</li><li>`startswith`&mdash;Checks if Checks if the `actual` value begins with the `expected` value.</li><li>`endswith`&mdash;Checks if the `actual` value ends with the `expected` value.</li><li>`equals`&mdash;Checks for numerical equality. For numerical comparison only.</li><li>`greater_than_or_equal`&mdash;Checks if the `actual` value is greater than or equal to (`>=`) the `expected` value. For numerical comparison only.</li><li>`greater_than`&mdash;Checks if the `actual` value is greater than (`>`) the `expected` value. For numerical comparison only.</li><li>`less_than`&mdash;Checks if the `actual` value is less than (`<`) the `expected` value. For numerical comparison only.</li><li>`less_than_or_equal`&mdash;Checks if the `actual` value is less than or equal to (`<=`) the `expected` value. For numerical comparison only.</li></ul><li>`actual`&mdash;Represents the string or numeric value obtained during the test run that you want to evaluate.</li><li>`expected`&mdash;The target string or numeric value to compare against the `actual` value. This is the same as `expectedValue` for out-of-the-box metrics.</li></ul> |
| `value`       | string  | Required for custom evaluation criteria. The value for the parameter. This field can be a literal value or a JSONPath expression if `isReference` is true. Typically, JSONPath expressions are used to dynamically retrieve `actual` parameters.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `isReference` | boolean | If true, indicates that `value` is a JSONPath expression referencing runtime data from the `generatedData` object returned by the [Get Test Results](https://developer.salesforce.com/docs/ai/agentforce/references/testing-api/testing-connect-reference.html#get-test-results) resource. If true, the `value` must be a JSONPath string. The default value is `false`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

#### AiEvaluationAgentTestCaseInput

Represents the input for a test case.

| Field Name            | Type                                                              | Description                                                      |
| :-------------------- | :---------------------------------------------------------------- | :--------------------------------------------------------------- |
| `utterance`           | string                                                            | Required. The request sent to the agent.                         |
| `contextVariable`     | [AiEvalCopilotTestCaseCntxtVar[]](#aievalcopilottestcasecntxtvar) | An XML array of context variables sent to the agent.             |
| `conversationHistory` | [AiEvalCopilotTestCaseConv[]](#aievalcopilottestcaseconv)         | An XML array of conversation history elements sent to the agent. |

##### AiEvalCopilotTestCaseCntxtVar

An XML array of context variables sent to the agent.

| Field Name      | Type   | Description                                  |
| :-------------- | :----- | :------------------------------------------- |
| `variableName`  | string | Required. The name of the context variable.  |
| `variableValue` | string | Required. The value of the context variable. |

##### AiEvalCopilotTestCaseConv

An XML array of conversation history sent to the agent.

| Field Name | Type    | Description                                                                                                                       |
| :--------- | :------ | :-------------------------------------------------------------------------------------------------------------------------------- |
| `role`     | string  | The role associated with a message. Valid values are `user` or `agent`. A conversation must begin with a message from the `user`. |
| `message`  | string  | The text from the user or agent.                                                                                                  |
| `topic`    | string  | Required for agent messages. Represents the subagent the agent used to generate a response.                                          |
| `index`    | integer | A zero based index for this conversation message.                                                                                 |

### Declarative Metadata Sample Definition

Here's an example of an AiEvaluationDefinition component.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<AiEvaluationDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
    <description>My Sample Tests</description>
    <name>my_test_n1</name>
    <subjectName>Agentforce_for_Salesforce</subjectName>
    <subjectType>AGENT</subjectType>
    <subjectVersion>v1</subjectVersion>
    <testCase>
        <number>1</number>
        <inputs>
          <utterance>Summarize the Global Media account</utterance>
        </inputs>
        <expectation>
            <name>topic_sequence_match</name>
            <expectedValue>OOTBSingleRecordSummary</expectedValue>
        </expectation>
        <expectation>
            <name>action_sequence_match</name>
            <expectedValue>['IdentifyRecordByName']</expectedValue>
        </expectation>
        <expectation>
            <name>bot_response_rating</name>
            <expectedValue>Summarization of the Global Media account</expectedValue>
        </expectation>
        <expectation>
            <name>conciseness</name>
        </expectation>
    </testCase>
    <testCase>
        <number>2</number>
        <inputs>
          <utterance>give me a pizza recipe</utterance>
        </inputs>
        <expectation>
            <name>topic_sequence_match</name>
            <expectedValue>Small_Talk</expectedValue>
        </expectation>
        <expectation>
            <name>action_sequence_match</name>
            <expectedValue>[]</expectedValue>
        </expectation>
        <expectation>
            <name>bot_response_rating</name>
            <expectedValue>the agent cannot answer this</expectedValue>
        </expectation>
    </testCase>
</AiEvaluationDefinition>
```

### Wildcard Support in the Manifest File

This metadata type supports the wildcard character \* (asterisk) in the `package.xml` manifest file. For information about using the manifest file, see [Deploying and Retrieving Metadata with the Zip File](https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/file_based_zip_file.htm).