--- schema_version: "1" id: 1b10b5b7eb5b5e4b106978ad8d92f25ff4d06cb6ead187b08aa33dd1933bd5c6 title: A2A Quality Evaluation Policy collection: mulesoft version: current locale: en-us portal_url: https://docs.salesforce.com/mulesoft/current/en-us/gateway/1.13/policies-included-a2a-quality-evaluation.md canonical_url: https://docs.mulesoft.com/gateway/1.13/policies-included-a2a-quality-evaluation source_site: docs.mulesoft.com content_type: documentation content_hash: 0fa9d0e755b7d64fdb513fbf8e8b025a81da8e133360783e2b191bccd7a46f21 description: MuleSoft Documentation Site release: "1.13" latest: false --- --- product: Omni Gateway version: 1.13 keywords: api gateway, flex gateway, gateway, policy --- # A2A Quality Evaluation Policy > For the full documentation index, see: https://docs.mulesoft.com/llms.txt

Policy Name	A2A Quality Evaluation
Summary	Evaluates A2A agent task responses with an LLM to score quality and completeness
Category	A2A
First Omni Gateway version available	v1.13.0
Release Notes	A2A Quality Evaluation Policy
Returned Status Codes	No return codes exist for this policy.

> [!NOTE] This policy supports Agent2Agent Protocol (A2A) version v0.3.0. To learn more about A2A > versions, see [A2A Releases](https://github.com/a2aproject/A2A/releases). ## Summary The A2A Quality Evaluation policy evaluates completed A2A agent task responses using an LLM to score completeness against the original user intent. The policy only adds its verdict as response headers and emits metrics for observability. The policy doesn’t block or modify the agent’s response. The policy evaluates quality when the final agent response is available: - **Synchronous responses** - When `message/send` returns a `200` status with the final response, the quality LLM is invoked inline before the response is returned. - **Asynchronous responses** - When `message/send` returns a `202` status with a task ID, the policy stores the task metadata. Later, when `tasks/get` returns a `completed` state, the policy invokes the quality LLM and caches the verdict for subsequent polls. > [!NOTE] This policy supports A2A `message/send` and `tasks/get` requests across all transport > protocols. ## Configuring Policy Parameters ### Omni Gateway Local Mode The A2A Quality Evaluation policy isn’t supported in Local Mode. ### Managed Omni Gateway and Omni Gateway Connected Mode When you apply the policy from the UI, the following parameters are displayed:

Element	Description	Required	Default
Quality LLM URL	OpenAI-compatible chat completions endpoint for quality evaluation. The policy POSTs request to `<qualityLlmUrl>/v1/chat/completions`. For example: `https://api.openai.com`	Yes	None
Quality LLM API Key	Bearer token for authenticating to the quality LLM endpoint. This value is encrypted at rest.	Yes	None
Quality LLM Model	Model name to use for quality evaluation. The name must be a valid model name supported by the configured endpoint. For example: `gpt-4o` or `gpt-4o-mini`	No	`gpt-4o`
Request Timeout (ms)	Timeout in milliseconds for the quality LLM call. If the call exceeds this timeout, the verdict isn’t attached and the policy logs a warning.	No	`10000`

## How This Policy Works The A2A Quality Evaluation policy intercepts A2A requests and evaluates completed agent responses using a separate LLM. The policy works differently for synchronous and asynchronous agent responses. ### Request Filtering The policy only processes requests that meet all of the following criteria: - HTTP method is `POST` - `Content-Type` header matches a JSON MIME type - Request body parses as a valid JSON-RPC request - JSON-RPC `method` is either `message/send` or `tasks/get` All other requests pass through without evaluation. ### Synchronous Quality Flow (`message/send` with `200`) When an agent returns a synchronous `200` response to `message/send`: 1. The policy captures the original user message and the agent’s final response. 2. The policy invokes the quality LLM with a prompt containing both the user message and agent response. 3. The quality LLM returns a verdict with three fields: - `evaluation`: `"yes"` or `"no"` (whether the agent fully addressed the user’s request) - `value`: `"1"`, `"2"`, or `"3"` (quality score: 1=poor, 2=adequate, 3=excellent) - `explanation`: Free-text explanation of the verdict 4. The policy adds quality verdict headers to the response. For header details, see [Quality Verdict Headers](#quality-verdict-headers). 5. The policy emits a `mulesoft_agent_calls` counter metric with the verdict tags (see [Observability Metrics](#observability-metrics)). 6. The response is returned to the client with the original agent response body and added quality headers. The quality LLM call happens inline, so it adds latency (up to the configured timeout) to the synchronous response path. ### Asynchronous Quality Flow (`message/send` with `202`, then `tasks/get`) When an agent returns an asynchronous `202` response to `message/send`: 1. The policy captures the original user message and the returned task ID. 2. The policy stores the task metadata. 3. The response is returned immediately to the client with no quality headers and no LLM call. Later, when a client polls with `tasks/get` and the upstream returns `state: "completed"`: 1. The policy invokes the quality LLM, stores the verdict, adds quality verdict headers, and emits the observability metric. 2. The response is returned with the quality headers. Subsequent polls of the same completed task return the stored verdict without re-invoking the LLM. ### Quality Verdict Headers When a quality verdict is available, the policy adds these headers to the response:

Header	Description
`x-quality-evaluation`	`"yes"` or `"no"` indicating whether the agent fully addressed the user’s request.
`x-quality-score`	Quality score: `"1"` (poor), `"2"` (adequate), or `"3"` (excellent).
`x-quality-explanation`	Free-text explanation of the verdict provided by the quality LLM.

### Observability Metrics Every time the quality LLM is invoked (synchronous `200` responses and the first `tasks/get` request that evaluates a completed task), the policy emits a single increment of the `mulesoft_agent_calls` counter metric. The metric includes these tags:

Tag	Values
`mulesoft.a2a.task_resolution`	`"yes"`, `"no"` (verdict from the quality LLM), or `"unknown"` if the LLM call failed or the response couldn’t be parsed.
`mulesoft.a2a.quality_score`	`"1"`, `"2"`, or `"3"` (score from the quality LLM), or `"unknown"` if the LLM call failed.
`mulesoft.a2a.method`	`"message/send"` or `"tasks/get"` (the A2A method that triggered the evaluation).
`mulesoft.a2a.latency_bucket`	One of 20 bounded log-scale labels representing the quality LLM call latency in milliseconds: `"le_10"`, `"le_15"`, `"le_25"`, `"le_40"`, `"le_60"`, `"le_90"`, `"le_130"`, `"le_200"`, `"le_300"`, `"le_450"`, `"le_700"`, `"le_1000"`, `"le_1500"`, `"le_2200"`, `"le_3300"`, `"le_5000"`, `"le_7500"`, `"le_11000"`, `"le_16000"`, or `"gt_16000"`. Only emitted on successful LLM calls.
`mulesoft.a2a.failure_reason`	Only emitted when `task_resolution="unknown"`. Possible values: `"transport_error"`, `"http_<code>"`, `"parse_error"`, `"no_choices"`, or `"invalid_evaluation"`.

> [!IMPORTANT] The metric counts distinct quality LLM calls, not distinct tasks. A task answered > synchronously via `200` produces one metric increment. A task accepted as `202` and later polled > to `completed` also produces one metric increment (on the first `tasks/get` poll that evaluates > the completed task). Subsequent polls against an already-evaluated task don’t emit additional > metrics. A sustained non-zero rate of `task_resolution="unknown"` indicates that the quality LLM is unreachable or returning errors. Operators should alert on this condition. ## Failure Modes The policy is designed to fail open — quality evaluation failures do not block agent responses. Instead, failures are logged and reflected in observability metrics.

Failure	Impact
Quality LLM unreachable, non-200 response, or parse error	No quality verdict headers are attached. The metric is emitted with `task_resolution="unknown"`, `quality_score="unknown"`, and an appropriate `failure_reason` tag. A warning is logged.
Verdict storage fails	The storage error is ignored. The policy may re-invoke the quality LLM on a subsequent `tasks/get` request. Both LLM calls are counted in the metric.
Request body is not valid JSON-RPC	The request passes through unchanged. No quality evaluation is performed.
Quality LLM call exceeds the configured timeout	Treated as a transport error. No quality verdict headers are attached. The metric is emitted with `task_resolution="unknown"` and `failure_reason="transport_error"`.

## Constraints and Limitations - **Synchronous latency** - For synchronous `200` responses, the quality LLM call happens inline and can add up to the configured timeout in latency. - **Asynchronous poll latency** - For asynchronous tasks, the first `tasks/get` poll that returns a `completed` state incurs the quality LLM call latency. ## Configuration Examples The following examples demonstrate common configurations for the A2A Quality Evaluation policy using different LLM models, timeouts, and endpoint configurations. ### Basic Configuration This example configures the policy to evaluate agent responses using the `gpt-4o` model with a 10-second timeout: ```yaml apiVersion: gateway.mulesoft.com/v1alpha1 kind: PolicyBinding metadata: name: a2a-quality-evaluation spec: targetRef: kind: ApiInstance name: my-a2a-agent policyRef: name: a-two-a-quality-evaluation-flex config: qualityLlmUrl: "https://api.openai.com" qualityLlmApiKey: "sk-proj-..." qualityLlmModel: "gpt-4o" timeout: 10000 ``` ### Custom Model and Timeout This example uses a faster model (`gpt-4o-mini`) with a shorter timeout for lower latency: ```yaml apiVersion: gateway.mulesoft.com/v1alpha1 kind: PolicyBinding metadata: name: a2a-quality-evaluation-fast spec: targetRef: kind: ApiInstance name: my-a2a-agent policyRef: name: a-two-a-quality-evaluation-flex config: qualityLlmUrl: "https://api.openai.com" qualityLlmApiKey: "sk-proj-..." qualityLlmModel: "gpt-4o-mini" timeout: 5000 ``` ### Using a Custom OpenAI-Compatible Endpoint This example uses a self-hosted or alternative OpenAI-compatible endpoint: ```yaml apiVersion: gateway.mulesoft.com/v1alpha1 kind: PolicyBinding metadata: name: a2a-quality-evaluation-custom spec: targetRef: kind: ApiInstance name: my-a2a-agent policyRef: name: a-two-a-quality-evaluation-flex config: qualityLlmUrl: "https://custom-llm.example.com" qualityLlmApiKey: "custom-api-key" qualityLlmModel: "custom-model-v1" timeout: 15000 ``` ## Sample Response Headers The following examples demonstrate the quality evaluation headers and metrics added by the policy for different response scenarios. ### Synchronous Quality Flow When an agent returns a synchronous `200` response, the policy evaluates the response inline and adds quality headers. 1. The initial request is sent: ```json { "jsonrpc": "2.0", "id": 1, "method": "message/send", "params": { "message": { "parts": [ { "kind": "text", "text": "Cancel my order #12345" } ] } } } ``` 2. The upstream agent returns `200` with the agent’s reply. The policy invokes the quality LLM, which returns: ```json { "evaluation": "yes", "value": "3", "explanation": "Order cancellation confirmed. The agent successfully canceled order #12345." } ``` 3. The policy adds these headers and emits these metrics: - Response headers: x-quality-evaluation: yes x-quality-score: 3 x-quality-explanation: Order cancellation confirmed. The agent successfully canceled order #12345. - Metric emitted: mulesoft\_agent\_calls{ task\_resolution="yes", quality\_score="3", method="message/send", latency\_bucket="le\_300" } 1 ### Asynchronous Quality Flow When an agent returns a `202` response, the policy stores the task metadata and defers evaluation until the task is polled. 1. Initial request is sent: ```json { "jsonrpc": "2.0", "id": 1, "method": "message/send", "params": { "message": { "parts": [ { "kind": "text", "text": "Cancel my order #12345" } ] } } } ``` 2. The upstream agent returns `202`: ```json { "jsonrpc": "2.0", "id": 1, "result": { "id": "task-abc-123" } } ``` The policy stores the task metadata. No quality headers are added. No metric is emitted. 3. The client polls the task: ```json { "jsonrpc": "2.0", "id": 2, "method": "tasks/get", "params": { "id": "task-abc-123" } } ``` 4. The upstream agent returns `200` with completed status: ```json { "jsonrpc": "2.0", "id": 2, "result": { "status": "completed", "message": { "parts": [ { "kind": "text", "text": "Your order #12345 has been canceled." } ] } } } ``` 5. The policy invokes the quality LLM, stores the verdict, and adds these headers and emits these metrics: - Response headers: x-quality-evaluation: yes x-quality-score: 3 x-quality-explanation: Order cancellation confirmed. - Metric emitted: mulesoft\_agent\_calls{ task\_resolution="yes", quality\_score="3", method="tasks/get", latency\_bucket="le\_450" } 1 Subsequent polls of the same task return the stored verdict and do not invoke the LLM or emit additional metrics. ## See Also - [A2A Prompt Decorator Policy](https://docs.mulesoft.com/gateway/1.13/policies-included-a2a-prompt-decorator) - [A2A Personally Identifiable Information (PII) Detector Policy](https://docs.mulesoft.com/gateway/1.13/policies-included-a2a-pii-detector) - [A2A Token Based Rate Limit Policy](https://docs.mulesoft.com/gateway/1.13/policies-included-a2a-token-rate-limit)