Evaluate how accurately your bot predicts intents by testing multiple utterances in bulk instead of one at a time. This method helps assess intent accuracy and identify gaps in intent configuration. Bulk testing helps identify misclassified utterances and assess overall model performance by using prediction metrics. Previously, testing utterances required manual, one-by-one validation. Bulk testing streamlines the process by analyzing large datasets and returning a consolidated prediction summary.
Required Editions
| View supported editions. |
| User Permissions Needed | |
|---|---|
| To build and manage Einstein Bots: |
Customize Application OR Modify Metadata OR Manage Bots |
Use bulk testing to:
- Evaluate up to 1000 utterances in a single request
- Compare expected and predicted intents
- Review confidence scores for each prediction
- Analyze overall performance by using summary metrics
You can perform bulk testing in two ways:
- Test Custom Utterances: Provide a dataset of utterances and expected intents to evaluate specific scenarios.
- Test Using Existing Bot Data: Evaluate all utterances already configured in the bot to generate a complete performance summary.
These two approaches correspond to different API endpoints.
Before you begin, turn on Einstein Bots and Einstein for Intents, and build your model. Turn on the Batch Input Tester pilot permission, and turn on the cross-lingual intent model (Intent V3) on the Bot Overview page.
How Bulk Testing Works
Bulk testing evaluates utterances against your bot's intent model and returns:
- You initiate an evaluation request with a set of utterances or a selected bot version.
- The intent model processes each utterance and predicts the most relevant intent.
- The system compares the predicted intent with the expected intent, when provided.
- The system assigns a confidence score and rating to each prediction.
- The system generates a prediction summary with overall performance metrics.
- The system returns consolidated results for all evaluated utterances.
Understand Prediction Results
To understand how accurately the bot classifies utterances and to identify areas for improvement, review the prediction results.
Per-Utterance Results. For each utterance, the system returns:
- Predicted intent: The intent identified by the model
- Prediction outcome: Indicates whether the predicted intent matches the expected intent (pass, fail, or unprocessed)
- Confidence score: Indicates how likely the prediction is correct
Summary Metrics. The system provides aggregated metrics to help assess overall performance:
- Success rate: Percentage of utterances predicted correctly
- Fail count: Number of utterances predicted incorrectly
- Intent coverage rate: Percentage of intents correctly predicted for at least one utterance
- Error count: Number of utterances that the bot couldn't process
Considerations
Keep these considerations in mind when you use bulk testing:
- You can evaluate a maximum of 1000 utterances at a time.
- Results depend on the quality of intent training data.
- Review failed predictions to improve intent accuracy.
- Errors for individual utterances don't stop the overall evaluation.
For API details, see Bot Utterance Intent Prediction Resources in the Connect REST API Developer Guide.