--- schema_version: "1" id: 1c5e1933be037049d48d9d1b320f5681ea6a1d6d28dcba3d9d77707fba64570c title: AiEvaluationDefinition collection: legacydeveloper version: current locale: en-us portal_url: https://docs.salesforce.com/legacydeveloper/current/en-us/api_meta/meta_aievaluationdefinition.md canonical_url: https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_aievaluationdefinition.htm source_site: developer.salesforce.com content_type: documentation content_hash: c5b12c8dd83b3663fbf508fb3b6c1a77859c3eb31508f978fd243d179008496f description: Represents an agent evaluation, including subject metadata and a set of test cases. --- # AiEvaluationDefinition Represents an agent evaluation, including subject metadata and a set of test cases. In Metadata API, you can create test definitions, including specifying inputs and expected outcomes, and deploy them to different orgs. In Connect API, you can execute test scenarios, poll for results, and retrieve test outcomes. This type extends the [Metadata](https://developer.salesforce.com/docs/atlas.en-us.264.0.api_meta.meta/api_meta/metadata.htm) metadata type and inherits its fullName field. For more information on testing agents, see the [Testing API Developer Guide](https://developer.salesforce.com/docs/einstein/genai/guide/testing-api.html). ## File Suffix and Directory Location AiEvaluationDefinition components have the suffix .aiEvaluationDefinition and are stored in the aiEvaluationDefinitions folder. ## Version AiEvaluationDefinition is available in API version 63.0 and later. Individual fields may have specific minimum API version requirements as noted in the field descriptions. ## Special Access Rules AiEvaluationDefinition is available only if Agentforce is enabled. See [Set Up Agents](https://help.salesforce.com/s/articleView?id=ai.copilot_setup.htm&language=en_US) in Salesforce Help. ## Fields | Field Name | Description | | --- | --- | | description | string Description The purpose of the test. | | name | string Description Required. The API name of the test. Can contain only underscores and alphanumeric characters and must be unique in your org. It must begin with a letter, not include spaces, not end with an underscore, and not contain two consecutive underscores. | | subjectName | string Description Required. A unique identifier for the agent being tested. Make sure that this identifier matches the API name of the agent, which you can find on the agent details page in Setup. | | subjectType | string Description Required. The type of subject being tested. The only currently supported value is AGENT. | | subjectVersion | string Description The agent version to test. If not provided, the latest active version is used by default. You can find the version in the BotVersion metadata type. | | testCase | [AiEvaluationTestCase\[\]](#AiEvaluationTestCase) Description A list of test cases. | ## AiEvaluationTestCase Represents a test case. | Field Name | Description | | --- | --- | | expectation | [AiEvaluationExpectation\[\]](#AiEvaluationExpectation) Description The criteria used to test the artifact's responses. | | inputs | [AiEvaluationAgentTestCaseInput\[\]](#AiEvaluationAgentTestCaseInput) Description The specific input provided to the artifact being tested. | | number | int Description The unique number for the test case. If not provided, the value is automatically calculated. | ## AiEvaluationExpectation Represents the expected outcome for a test case. | Field Name | Description | | --- | --- | | expectedValue | string Description The expected outcome of the test. The format of this field depends on the value of the name field. The expected outcome is compared against the response generated when you run the test using [Connect REST API](https://developer.salesforce.com/docs/einstein/genai/guide/testing-api-connect.html) | | label | string Description An optional label for an expectation. Typically added when using the same custom expectation name multiple times in a test case. If provided, this label appears in the test results; otherwise, the expectation name appears. | | name | string Description Required. The expectation name. Valid values are: - topic\_sequence\_match: The expectedValue field value is a string representing the topic that the agent is expected to use, such as OOTBSingleRecordSummary. For a list of agent topics, see Standard Agent Topic Reference in Salesforce Help. - action\_sequence\_match: The expectedValue field value is a string\[\] representing a list of actions that you expect the artifact to take during the test, such as \['IdentifyRecordByName', 'action2'\]. For a list of agent actions, see Standard Agent Action Reference in Salesforce Help. This option was previously called action\_sequence\_match. - bot\_response\_rating: The expectedValue field value is a string representing the expected response generated by the artifact, such as Summarization of the Global Media account. - coherence: A generated answer is coherent if it’s easy to understand and has no grammatical errors. If you use this quality check, you don't need an expectedValue field value. - completeness: A generated answer is complete if it includes all the essential information. If you use this quality check, you don't need an expectedValue field value. - conciseness: A generated answer is concise if it's brief but comprehensive. Shorter is better. If you use this quality check, you don't need an expectedValue field value. - output\_latency\_milliseconds: Latency in milliseconds from sending a request until a response is received. If you use this quality check, you don't need an expectedValue field value. - string\_comparison: A custom evaluation criteria that tests a response for a specified string value. - numeric\_comparison: A custom evaluation criteria that tests a response for a specified numeric value. | | parameter | [AiEvaluationTestCaseCritParam\[\]](#AiEvaluationTestCaseCritParam) Description Required for custom test criteria. An array of parameters for the specific custom criteria defined by expectation.name. This field replaces expectedValue for custom test criteria. | ## AiEvaluationTestCaseCritParam Defines a criterion parameter for expectations, including name, value, and whether it references another value. Available in API version 64.0 and later. | Field Name | Description | | --- | --- | | isReference | boolean Description If true, indicates that value is a JSONPath expression referencing runtime data from the generatedData object returned by the Get Test Results resource. If true, the value must be a JSONPath string. The default value is false. | | name | string Description Required for custom evaluation criteria. The name of the parameter required by the evaluation. Valid values are: operator—type of comparison; actual—runtime value to evaluate; expected—target value to compare against. For operator, valid options include: - equals: Checks if the actual value exactly matches the expected value (string or numeric). - contains: Checks if the actual string contains the expected string. - startswith: Checks if the actual string begins with the expected string. - endswith: Checks if the actual string ends with the expected string. - greater\_than\_or\_equal: Checks if the numeric actual value is greater than or equal to the numeric expected value (\>=). - greater\_than: Checks if the numeric actual value is greater than the numeric expected value (\>). - less\_than: Checks if the numeric actual value is less than the numeric expected value (<). - less\_than\_or\_equal: Checks if the numeric actual value is less than or equal to the numeric expected value (<=). | | value | string Description Required for custom evaluation criteria. The value for the parameter. This field can be a literal value or a JSONPath expression if isReference is true. Typically, JSONPath expressions are used to dynamically retrieve actual parameters. | ## AiEvaluationAgentTestCaseInput Represents the inputs for a test case, including variables, conversation history, and the utterance. | Field Name | Description | | --- | --- | | contextVariable | [AiEvalCopilotTestCaseCntxtVar\[\]](#AiEvalCopilotTestCaseCntxtVar) Description An XML array of context variables sent to the agent. | | conversationHistory | [AiEvalCopilotTestCaseConv\[\]](#AiEvalCopilotTestCaseConv) Description An XML array of conversation history elements sent to the agent. | | utterance | string Description Required. The request sent to the agent. | ## AiEvalCopilotTestCaseCntxtVar An XML array of context variables sent to the agent. | Field Name | Description | | --- | --- | | variableName | string Description Required. The name of the context variable. | | variableValue | string Description Required. The value of the context variable. | ## AiEvalCopilotTestCaseConv An XML array of conversation history sent to the agent. | Field Name | Description | | --- | --- | | index | integer Description A zero based index for this conversation message. | | message | string Description The text from the user or agent. | | role | string Description The role associated with a message. Valid values are user or agent. A conversation must begin with a message from the user. | | topic | string Description Required for agent messages. Represents the topic the agent used to generate a response. | ## Declarative Metadata Sample Definition Here's an example of an AiEvaluationDefinition component. My Sample Tests my\_test\_n1 Agentforce\_for\_Salesforce AGENT v1 1 Summarize the Global Media account topic\_sequence\_match OOTBSingleRecordSummary action\_sequence\_match \['IdentifyRecordByName'\] bot\_response\_rating Summarization of the Global Media account conciseness 2 give me a pizza recipe topic\_sequence\_match Small\_Talk action\_sequence\_match \[\] bot\_response\_rating the agent cannot answer this ## Wildcard Support in the Manifest File This metadata type supports the wildcard character \* (asterisk) in the package.xml manifest file. For information about using the manifest file, see [Deploying and Retrieving Metadata with the Zip File](https://developer.salesforce.com/docs/atlas.en-us.264.0.api_meta.meta/api_meta/file_based_zip_file.htm).