Fine-tune and validate retrieval logic for retrieval-augmented generation (RAG) systems by using retriever playground, reducing deployment risks, and verifying accurate AI outputs. Use AI Models (formerly Einstein Studio) retriever playground to configure pre-filters, select test data, and adjust result parameters to customize retrieval logic for domain-specific use cases, such as retrieving complaince documents or knowledge articles.
Required Editions
| Available in: All Editions supported by Data 360. See Data 360 edition availability. |
Review intermediate outputs and key metrics, such as the Source Recall Score or Context Recall Score, to validate that the retriever finds information from expected source documents. Fine-tuning retrieval logic in the retriever playground is a cost-effective way to test and debug retrievers. Retriever playground improves agent performance and search relevance, establishing reliable and meaningful generative AI outcomes.
Use Case: Agent Answers Field Service Questions on Installing Double Door Refrigerators with Incomplete Results
A business administrator sets up an agent to respond to questions from contractors in the field who install their company refrigerators.
Problem: Although the company knowledge base includes an article with complete step-by-step installation instructions, the field service provides feedback that responses from the agent are incomplete. The business admin tests the retriever set up for the agent in the retriever playground.
The test shows that only five steps are retrieved from the eleven-step article.
In reviewing the retriever results panel, the admin notices that the search index splits the article into two chunks. The top chunk includes the "double-door refrigerator" keyword. Because the second chunk dosen't include the necessary keywords from the question, it's scored lower in the result set and doesn't appear in the response.
Solution: The admin increases the number of test results to include the chunk that contains the rest of the instructions. Then, the admin can test the new configuration with sample questions about installing double-door refrigerators to make sure that the response is complete.
- Test an Individual Retriever
Test a retriever configuration to ensure accurate responses from your AI agents. Save retriever configurations as new versions of the retriever in AI Models (formerly Einstein Studio). Activate them for use in agents with prompt templates, Flow, and apps. - Test an Ensemble Retriever
Test an ensemble retriever configuration in the AI Models retriever playground before you use it in prompt templates or AI agents. Add or remove retrievers to an ensemble, configure test data, and result parameters to verify the retrieval logic. - Retriever Metrics and Results
Review retriever metrics and the retrieved data chunks to understand the results of your test. Modify the retriever configuration to try different configurations and expected source data to get an optimal response. Save your configuration as a new version of the retriever. - Billing Considerations for Testing Retrievers
When you test retrievers by querying search indexes, your data is processed in Data 360. The primary usage type for retriever testing is Data Queries. Use of retriever playground impacts the consumption of credits used for billing in the following usage types.