---
schema_version: "1"
id: ce7abfb6b869432dee3e212ecbc420f27710efc44c66c9c27686ec5f169de46a
title: Toxicity Detection
collection: admin
version: current
locale: en-us
portal_url: https://docs.salesforce.com/admin/current/en-us/ai/264-0-0/generative_ai_trust_toxicity_detection.md
canonical_url: https://help.salesforce.com/s/articleView?id=ai.generative_ai_trust_toxicity_detection.htm&release=264.0.0&type=5
source_site: help.salesforce.com
content_type: documentation
content_hash: 911a94362eb8e40941f18f3ab8644eaa916f0f5c34f49247359e422cb68782c6
description: Einstein Trust Layer uses machine learning (ML) models to identify and flag toxic content in prompts and responses.
products:
  - Agentforce
  - Cross Cloud
release: "264"
---
# Toxicity Detection

Einstein Trust Layer uses machine learning (ML) models to identify and flag toxic content in prompts and responses.

### Required Editions

<table class="slds-table slds-table_bordered slds-max-medium-table_stacked slds-m-bottom_small edition"><colgroup><col></colgroup><tbody><tr class=""><td class="slds-cell-wrap" style="vertical-align:top;">Available in: <strong>Enterprise</strong>, <strong>Performance</strong>, and <strong>Unlimited</strong> Editions with an Einstein for Sales, Einstein for Platform, Einstein for Service, Einstein 1 Service, or Einstein GPT Service add-on. To purchase add-ons, contact your Salesforce account executive.</td></tr></tbody></table>

Customer-facing outputs from your AI applications represent your company’s brand and voice. AI can sometimes generate toxic or harmful content that can lead to reputation harm to your company. Toxicity in responses can also be influenced by prompts, so it’s also important to detect toxicity in prompts as well as in responses. Toxicity in prompts can come from untrusted sources such as public chat interactions and third-party web content.

![Note](https://sf-zdocs-cdn-prod.zoominsoftware.com/tdta-ai-generative_ai-264-0-0-production-enus/f8ef6498-d7b2-44e3-9b34-ffd5a850aae9/images/icon_note.png)

Note Toxicity Detection in responses is enabled by default and can’t be changed. Toxicity Detection in prompts (beta) is turned off by default, but you can enable it for your Salesforce org.

When toxicity is detected in prompts or responses, you see a notification or a warning in the Salesforce AI features at run time. For example, you see toxicity warnings in copilot or prompt builder if toxic content is detected in the generated response from the LLM.

![Note](https://sf-zdocs-cdn-prod.zoominsoftware.com/tdta-ai-generative_ai-264-0-0-production-enus/f8ef6498-d7b2-44e3-9b34-ffd5a850aae9/images/icon_note.png)

Note Disclaimer: Toxicity warning isn't available in all AI features.

**Toxicity Warning in Prompt Builder**

![The image displays a warning message, aleting you that the content is toxic and potentially harmful](https://sf-zdocs-cdn-prod.zoominsoftware.com/tdta-ai-generative_ai-264-0-0-production-enus/f8ef6498-d7b2-44e3-9b34-ffd5a850aae9/generative_ai/images/trust_layer_toxicity_warning_in_prompt_builder.png)

![Important](https://sf-zdocs-cdn-prod.zoominsoftware.com/tdta-ai-generative_ai-264-0-0-production-enus/f8ef6498-d7b2-44e3-9b34-ffd5a850aae9/images/icon_important.png)

Important Although our toxicity detection models have shown to be effective during internal testing, it's important to note that no model can guarantee 100% accuracy. In addition, cross-region and multinational use cases can affect the ability to detect specific data patterns. With trust as our priority, we're dedicated to the ongoing evaluation and refinement of our models.

## Toxicity Categories

Einstein toxicity detection models recognize these categories:

 
| Category | Type of Content |
| --- | --- |
| Violence | Content that depicts, references, or incites behavior intended to cause physical harm to people, animals, or property |
| Sexual | Content that depicts, references, or solicits material, behavior, or language containing sexual language, imagery, or themes, including consensual and nonconsensual sexual content, illegal and legal sexual acts and behaviors, and sexually suggestive and flirtatious content |
| Profanity | Content that includes inflammatory, offensive, obscene, vulgar, or irreverent language, gestures, and expletives |
| Hate | Content that depicts, references, or incites behavior or language intended to cause psychological harm to a person or group on the basis of identity or other distinguishing personal traits |
| Physical | Content that depicts, references, encourages, or enables the use, acquisition, or distribution of illicit substances, nonprescription medication, and other substances that have a physiological or psychological effect when consumed or, behavior intended to cause physical harm, self-harm, or death |

## Toxicity Scoring

Each category of toxic content is rated to indicate the likelihood of that type of toxic language in the text. Additionally, the Einstein Trust Layer gives an overall toxicity score that reflects the combination of all detected categories.

The scores range from 0 to 1, with 1 being the most toxic. The scores are logged in an audit trail and stored in Data 360. The Trust Layer prebuilt reports and dashboards visualize toxicity trends in features and time. You can also create custom reports in Data 360.

#### See Also

- [Turn On Toxicity Detection For Prompts (Beta)](https://help.salesforce.com/s/articleView?id=ai.generative_ai_trust_configure_toxicity_detection.htm&release=264.0.0&type=5 "Turn on toxicity detection in prompts so you can monitor the prompts that have toxic language and modify them accordingly. These settings are applied to your Salesforce org.")