# developer Einstein Vision and Language (einstein-vision-language) - Full Documentation # Code Samples and Learning Resources ## Code Samples - [salesforce-einstein-platform-apex](https://github.com/muenzpraeger/salesforce-einstein-platform-apex)—GitHub repo that contains code for an Apex-based wrapper for both the Einstein Vision APIs and the Einstein Language APIs. - [salesforce-einstein-vision-java](https://github.com/muenzpraeger/salesforce-einstein-vision-java)—GitHub repo that contains code for a Java-based wrapper for the Einstein Image Classification API. - [salesforce-einstein-vision-swift](https://github.com/muenzpraeger/salesforce-einstein-vision-swift)—GitHub repo that contains code for a Swift-based wrapper for the Einstein Image Classification API. - [salesforce-einstein-object-detection-mobile-example](https://github.com/davescruggs/salesforce-einstein-object-detection-mobile-example)—GitHub repo that contains code for a sample mobile app that uses the Einstein Object Detection API. - [Einstein Vision Quick Start Code Samples](https://github.com/MetaMind/quickstart)—GitHub repo that contains Java and Ruby code samples that show you how to generate a JWT access token, refresh a token, and make a prediction using the Einstein Image Classification API. - [EinsteinVisionPython](https://github.com/feliperyan/EinsteinVisionPython)—GitHub repo that contains code for a Python-based wrapper for the Einstein Image Classification API. - [einstein-ai](https://github.com/mshanemc/einstein-ai)—GitHub repo that contains objects and code that you can use to access Einstein Vision and Einstein Language from Salesforce. - [einstein-scala-lib](https://github.com/rajdeepd/einstein-scala-lib)—GitHub repo that contains the code for a Scala-based wrapper for the Einstein Vision APIs. - [einstein-go-apis](https://github.com/rajdeepd/einstein-go-apis)—GitHub repo that contains the code for a Golang-based wrapper for the Einstein Vision and Einstein Language APIs. **Note:** Avoid security vulnerabilities and CVEs by using the latest versions of code libraries and dependencies in any sample code. ## Apps and Services - [Einstein Vision and Language Model Builder Open Source](https://github.com/SFDC-Assets/EinsteinPlayground)—The open source version of the Einstein Vision and Language Model Builder. - [Einstein Intent Assessor](https://appexchange.salesforce.com/appxListingDetail?listingId=a0N3A00000G5qvaUAB)—An Einstein Analytics template, available on AppExchange, that helps you visualize the accuracy of your Einstein Intent models. You can identify specific labels and utterances that your model classifies with less accuracy. # Community Sentiment Model Einstein Language offers a pre-built sentiment model that you can use as long as you have a valid JWT token. This model has three classes: - positive - negative - neutral Use the community sentiment model to classify text without building your own custom model. This model was created from data that comes from multiple sources. The data is short snippets of text, about one or two sentences, and similar to what you would find in a public community or Chatter group, a review/feedback forum, or enterprise social media. This cURL command sends in a text string and returns a prediction from the model. You call the pre-built model the same way you call a custom model, but instead of passing in your own `modelId`, you pass in a `modelId` of `CommunitySentiment`. See [Prediction for Sentiment](/docs/analytics/einstein-vision-language/references). ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "modelId=CommunitySentiment" -F "document=the presentation was great and I learned a lot" https://api.einstein.ai/v2/language/sentiment ``` The model returns a result similar to the following JSON. ```json { "probabilities": [ { "probability": 0.9435403, "label": "positive" }, { "probability": 0.031840604, "label": "negative" }, { "probability": 0.02461909, "label": "neutral" } ], "object": "predictresponse" } ``` # Content Takedown Salesforce Einstein respects the intellectual property rights of third parties and responds to allegations that copyrighted material has been posted, uploaded or shared on or through the machine learning-based classification and prediction service operated by Salesforce Einstein (the “Service”) without authorization from the copyright holder in accordance with the safe harbor set forth in the Digital Millennium Copyright Act (“DMCA”). Salesforce Einstein will also, in appropriate circumstances and at its discretion, disable and/or terminate the accounts of users who may infringe or repeatedly infringe the copyrights of others in accordance with the DMCA. ## Notification of Alleged Copyright Infringement If you believe that your work has been copied and made available through the Service in a way that constitutes copyright infringement, you may send a written document to Salesforce Einstein’s Designated Agent (as set forth below) that contains the following (a “Notice”): - A description of the copyrighted work that you claim has been infringed. - Identification of the URL or other specific location that contains the material that you claim infringes your copyright described in Item 1 above. You must provide us with reasonably sufficient information to locate the allegedly infringing material. - An electronic or physical signature of the owner of the copyright or of the person authorized to act on behalf of the owner of the copyright. - A statement by you that you have a good faith belief that the disputed use is not authorized by the copyright owner, its agent, or applicable law. - A statement by you that the information contained in your Notice is accurate and that you attest under the penalty of perjury that you are the copyright owner or that you are authorized to act on the owner's behalf. - Your name, mailing address, telephone number, and email address. Salesforce Einstein’s Designated Agent for Notice of claims of copyright infringement can be reached as follows: **Salesforce Einstein** 415 Mission Street, 3rd Floor San Francisco, California 94105 Attn: Legal Department, DMCA Complaint E-Mail: copyright@salesforce.com Please note that you may be liable for damages, including court costs and attorneys fees, if you misrepresent that content uploaded by a Service user is infringing your copyright. Upon receiving a proper Notice, Salesforce Einstein will remove or disable access to the allegedly infringing material and notify the alleged infringer of your claim. We will also advise the alleged infringer of the DMCA Counter Notice Procedure described below in Section B by which the alleged infringer may respond to your claim and request that we restore this material. ## Counter Notice Procedure If you believe your own copyrighted material has been removed from the Service in error, you may submit a written Counter Notice to our Designated Agent (as identified above) that includes the following: - Identification of the material that has been removed or disabled and the location at which the material appeared before it was removed or disabled. - A statement that you consent to the jurisdiction of the Federal District Court in which your address is located, or if your address is outside the United States, the Northern District of California or any other judicial district in which Salesforce Einstein may be found. - A statement that you will accept service of process from the party that filed the Notice or the party's agent. - Your name, address and telephone number. - A statement under penalty of perjury that you have a good faith belief that the material in question was removed or disabled as a result of mistake or misidentification of the material to be removed or disabled. - Your physical or electronic signature. If you send our Designated Agent a valid, written Counter Notice meeting the requirements described above, we will restore your removed or disabled material within 10 to 14 business days from the date we receive your Counter Notification, unless our Designated Agent first receives notice from the party filing the original Notice informing us that such party has filed a court action to restrain you from engaging in infringing activity related to the material in question. Please note that if you misrepresent that the disabled or removed content was removed by mistake or misidentification, you may be liable for damages, including costs and attorney's fees. ## Repeat Infringer Policy The Company may withdraw all rights and privileges relating to the Service from any user who is deemed to be a repeat infringer. This determination will be based on the number of “strikes” against the user. A “strike” is counted against a user each time there is either: (i) an adjudication by a court, arbitrator or other tribunal of competent jurisdiction that the user has engaged in copyright infringement of any kind in relation to the Service; or (ii) the Company has actual knowledge, regardless of any such adjudication, that the user has engaged in any such copyright infringement. Each adjudication or instance of knowledge counts as a separate strike. If an adjudication or instance of knowledge pertains to multiple instances of copyright infringement, it can count as multiple strikes. The Company has adopted a “three strikes and you’re out” policy under which a user who accumulates three strikes is considered a repeat infringer and may be subject to account termination. # Create a French Model In this walkthrough, you use the Einstein Intent API to create a French model to route support cases. You can then use the model to analyze text and infer what the user wants to accomplish. You use the French case routing [sample .csv file](https://einstein.ai/text/case_routing_intent_fr.csv) that we make available to you. ## Step by Step: Create a French Model ### Step 1: Get a Token A valid JWT token is required to make API calls. 1. Find your key file. If you already created an account, locate the einstein_platform.pem file that you downloaded as part of that process. This file contains your private key. 2. Install cURL. You use the cURL command-line tool throughout the following walkthrough. The tool is installed by default on Linux and OSX. If you don’t already have it installed, download it from [https://curl.haxx.se/download.html](https://curl.haxx.se/download.html). Alternatively, you can use Postman to make API calls. 3. Get a token. The Einstein Intent API uses the OAuth 2.0 JWT bearer token flow for authorization. Use the [token page](https://api.einstein.ai/token) to upload your key file and generate a JWT token. ### Step 2: Create a Dataset 1. Replace `` with your JWT token, and run the following command from the command line. This command: - Creates a dataset called Dataset de Routage de Cas from the specified .csv file by accessing the file via a URL. If you leave out the name parameter, the dataset name defaults to the name of the file: case_routing_intent_fr. - Creates five labels as specified in the .csv file. - Creates 150 examples. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Dataset de Routage de Cas" -F "path=http://einstein.ai/text/case_routing_intent_fr.csv" -F "language=fr" -F "type=text-intent" https://api.einstein.ai/v2/language/datasets/upload ``` This call is asynchronous, so the response looks like this JSON. ```json { "id": 1020389, "name": "Dataset de Routage de Cas", "createdAt": "2020-03-16T17:54:51.000+0000", "updatedAt": "2020-03-16T17:54:51.000+0000", "labelSummary": { "labels": [] }, "totalExamples": 0, "available": false, "statusMsg": "UPLOADING", "type": "text-intent", "language": "fr", "numOfDuplicates": 0, "object": "dataset" } ``` 2. To verify that the data has been loaded, make a call to get the dataset. Replace `` with your JWT token, and replace `` with the ID of the dataset that you created. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/datasets/ ``` The results look something like this JSON. You know the dataset is ready when `available` is `true` and `statusMsg` is `SUCCEEDED`. You can see that the response also returns the `language` field. ```json { "id": 1020389, "name": "Dataset de Routage de Cas", "createdAt": "2020-03-16T17:54:51.000+0000", "updatedAt": "2020-03-16T17:54:51.000+0000", "labelSummary": { "labels": [ { "id": 104866, "datasetId": 1020389, "name": "Informations d'Expédition", "numExamples": 30 }, { "id": 104867, "datasetId": 1020389, "name": "Facturation", "numExamples": 24 }, { "id": 104868, "datasetId": 1020389, "name": "Opportunité de Vente", "numExamples": 44 }, { "id": 104869, "datasetId": 1020389, "name": "Changement de Commande", "numExamples": 26 }, { "id": 104870, "datasetId": 1020389, "name": "Aide Mot de Passe", "numExamples": 25 } ] }, "totalExamples": 149, "totalLabels": 5, "available": true, "statusMsg": "SUCCEEDED", "type": "text-intent", "language": "fr", "numOfDuplicates": 1, "object": "dataset" } ``` ### Step 3: Train the Dataset to Create a Model 1. In the following command, replace `` with your JWT token, and replace `` with the ID of the dataset that you created, and run the command. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -F "name=Modèle de Routage de Cas" -F "algorithm=multilingual-intent" -F "datasetId=" https://api.einstein.ai/v2/language/train ``` The response looks like this JSON. ```json { "datasetId": 1020389, "datasetVersionId": 0, "name": "Modèle de Routage de Cas", "status": "QUEUED", "progress": 0, "createdAt": "2020-03-16T17:54:51.000+0000", "updatedAt": "2020-03-16T17:54:51.000+0000", "modelId": "EIN6TP2LJYUOAW44EH3VH2Z4DU", "learningRate": 0, "epochs": 0, "queuePosition": 1, "modelType": "text-intent", "algorithm": "multilingual-intent", "language": "fr", "object": "training", "trainParams": null, "trainStats": null } ``` 2. Use the `modelId` to make this call and get the training status. Replace `` with your JWT token, and replace `MODEL_ID` with the ID of the model that you created. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/train/ ``` The training status response looks like this JSON. When `status` is `SUCCEEDED`, it means the training was successful, and the model is ready for predictions. ```json { "datasetId": 1020389, "datasetVersionId": 15667, "name": "Modèle de Routage de Cas", "status": "SUCCEEDED", "progress": 1, "createdAt": "2020-03-16T17:54:51.000+0000", "updatedAt": "2020-03-16T17:54:51.000+0000", "modelId": "EIN6TP2LJYUOAW44EH3VH2Z4DU", "learningRate": 0, "epochs": 1000, "modelType": "text-intent", "algorithm": "multilingual-intent", "language": "fr", "object": "training", "trainParams": null, "trainStats": { "labels": 5, "examples": 149, "averageF1": 0.8880952380952379, "totalTime": "00:04:16:861", "transforms": null, "trainingTime": "00:04:16:280", "earlyStopping": true, "lastEpochDone": 64, "modelSaveTime": "00:00:00:262", "testSplitSize": 25, "trainSplitSize": 124, "datasetLoadTime": "00:00:00:581", "preProcessStats": null, "postProcessStats": null } } ``` ### Step 4: Send Text for Prediction Now your model is ready to go. To test it, send some text in for prediction. This cURL call takes the `modelId` of the model from which you want to return a prediction and the text string to analyze. In this case, the string in English is, “I would like to buy a soccer jersey.” 1. Replace `` with your JWT token, and replace `` with the ID of your model. Then run the command. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "modelId=" -F "document=Je voudrais acheter un maillot de foot" https://api.einstein.ai/v2/language/intent ``` The response looks like this JSON. The model predicts that the text indicates that the user is interested in buying something, so the model returns Opportunité de Vente (Sales Opportunity) as the top probability. Your app can then use this information to route the case to the correct department or sales person. ```json { "probabilities": [ { "probability": 0.95638543, "label": "Opportunité de Vente" }, { "probability": 0.02187803, "label": "Changement de Commande" }, { "probability": 0.01103622, "label": "Informations d'Expédition" }, { "probability": 0.008903132, "label": "Facturation" }, { "probability": 0.0017971398, "label": "Aide Mot de Passe" } ], "object": "predictresponse" } ``` # Data Science The terms “data science,” “machine learning,” and “artificial intelligence” are often used interchangeably, but they don’t all mean the same thing. **Data science** is a broad interdisciplinary field that uses automated, scientific methods to analyze large amounts of data and extract knowledge from that data. You can use these methods to: - Train the predictive models that are central to machine learning and artificial intelligence - Analyze, visualize, and transform the data fed into those models - Analyze the health and performance of models in production **Artificial intelligence (AI)** is an umbrella term that refers to teaching computers to perform complex tasks. The use of AI ranges from chatbots that field simple customer support questions to self-driving cars that intelligently navigate real-world traffic. AI behaves “like a human.” It can take complex, unpredictable input and respond with complex output that gives the appearance of agency. Often AI deals with the environment it’s placed in and makes various complicated decisions. Robots who play chess and the personal assistant on your phone? Those are examples of AI. **Machine learning** is a technique that takes in data and produces a refined, specific type of output. Machine learning helps a computer “learn” from examples without being explicitly programmed with step-by-step instructions. Unlike AI, which typically responds to a diverse stream of inputs and outputs, machine-learning algorithms are geared toward answering a single type of question well. We don’t expect these algorithms to perform exactly like humans. In fact, we generally expect them to answer certain questions better than a human. For that reason, machine-learning algorithms are at the forefront of efforts to diagnose diseases, predict stock market trends, and recommend music. Machine-learning techniques can help create AI, but the techniques aren’t sophisticated enough to constitute AI on their own. They can recognize and tag people in your photos or let your navigation app predict how long it takes to drive home at rush hour. In these examples, a computer takes in data, such as images that you already tagged of your friends or information from other drivers about how fast traffic is moving on the roads you need to take at 5 PM, and produces a specific output. Data science, artificial intelligence, and machine learning all overlap. A Venn diagram of these disciplines looks like this: ![MachineLearningVenn](https://developer.salesforce.com/media/machine_learning_venn.png) **Deep learning** is a type of machine learning. In deep learning, you create and train a neural network in a specific way. A neural network is a set of algorithms designed to recognize patterns. In deep learning, the neural network has multiple layers. At the top layer, the network trains on a specific set of features and then sends that information to the next layer. The network takes that information, combines it with other features and passes it to the next layer, and so on. Deep learning has increased in popularity because it has proven to outperform other methodologies for machine learning. Due to the advancement of distributed compute resources and businesses generating an influx of image, text, and voice data, deep learning can deliver insights that weren’t previously possible. ## How Machines Learn Unlike a traditional program, where every instruction for the computer to follow is laid out explicitly, step by step, machine learning lets the computer build its own steps. It does this by using data to build a model. A wide variety of machine learning algorithms are available, each suited to performing a specific task. For each machine learning task, a data scientist selects the right generic algorithm and chooses the right kind of data to feed into it. The algorithm looks for patterns in the data that map the input data to the desired output. Then the algorithm uses those patterns to build a mathematical model that describes the data. For example, say you’re designing a trip-planning app that helps consumers find the best time to buy a cheap plane ticket. First, you train a model to predict when airlines have the lowest price for a specific itinerary, given historical data on ticket pricing for the route, holidays, departure time, and how long in advance you buy the ticket. The model uses results from the past to predict what is likely to happen in the future. Then, when a customer enters the desired travel dates and route, you feed the model information about what the customer needs and get suggestions back about when the tickets are cheapest. Over time, you continue feeding data to the model, which adjusts itself, improving its suggestions as its dataset (and therefore its knowledge about pricing trends) grows. At a high level, designing a machine-learning solution involves these four steps. 1. Identify the outcome you want. For example, say that you own a clothing store. You want to bring in the maximum revenue from sales of your clothing. This outcome is the output of your algorithm. 2. Identify your inputs. Often, these inputs are the “levers” that affect your outcome. For the revenue of a clothing store, that can include price and in-store promotions. These items are the input for your algorithm. 3. Build a model that connects the input that you can control to the outcome you want. You can see how changes you make to your input affect the outcome. For example, you can see how changing the price for your products (inputs) affects your revenue (the outcome). 4. Optimize. Use your model to search for and find the best inputs to achieve the outcome you want. You can use this basic blueprint to design machine-learning solutions for a wide range of problems, such as: - Web search ranking - Autocomplete features - Credit card fraud detection - Address or handwriting recognition - Voice assistants - Language translation - Autonomous navigation (robotic vacuums, self-driving cars, space exploration vehicles) - Targeted ads based on your online activity - Facial recognition - Automatic product recommendations - Medical diagnostics - Spam filtering ## Machine Learning and Algorithms Data scientists use machine-learning models to extract information from data. But where do those models come from? A computer builds a model by taking a generic machine-learning algorithm and feeding it a set of training data. There are many machine-learning algorithms, and the broadest way to categorize them is by learning style. The two major learning styles are supervised and unsupervised. ### Supervised vs. Unsupervised The learning style of an algorithm has to do with the kind of data it uses to create a model. A supervised algorithm takes input data called “training data,” which has a known label or result. A data scientist knows what results this training data produces and can compare those known results to the actual results of the model and correct the model until it’s accurate. For example, if you want a supervised algorithm to produce a model that can identify pictures with cats, you feed it a set of labeled data containing pictures with cats and pictures without them. Because the data is labeled and you know which pictures are in each category, you can adjust the model until it reliably identifies which pictures contain cats and which don’t. An unsupervised algorithm takes in unlabeled data and creates a model by looking for patterns in that data. It can look for general patterns or rules in the data, or it can organize the data into groupings by similarity. For example, you can use an unsupervised model to examine customer data, such as purchase history and demographics, to categorize similar customers for specific marketing campaigns. ### Predictive vs. Descriptive You can also think about supervised and unsupervised models as either predictive or descriptive. A supervised model is usually predictive: It uses existing data to build a model that you can use to predict what can happen under specific conditions in the future. For example, a predictive model can predict whether changing prices increases sales and revenue. An unsupervised model is usually descriptive: It takes existing data and describes it by examining the rules and patterns in the data and categorizing or arranging it. For example, a descriptive model can tell you what kinds of customers you have by grouping them into segments based on age or spending habits. Both Einstein Vision and Einstein Language use supervised algorithms. Therefore, the models they create are predictive. ## Classification Machine-learning problems typically fall into four functional groups: classification, clustering, regression, and dimensionality reduction. ![AlgorithmsFunctionalGroups](https://developer.salesforce.com/media/algorithms_function.png) We’ll focus on classification, because Einstein Vision and Language builds classification models. A classification algorithm sorts data into categories and gives a result that’s a label, for example, “cat” or “not a cat.” Because a classification algorithm is supervised, you train it with a set of labeled data. Then you can check the accuracy of the resulting model against that data. ## The Language of Data Science The world of AI and deep learning has introduced many new terms. Understanding these terms and how they relate to each other makes it easier to work with Einstein Vision and Einstein Language. - Dataset—Contains the training data. The dataset is the starting point for an Einstein Vision or Language implementation because it contains the data from which the model is created. After you gather your data, whether it’s images or text, you use that data to create the dataset. - Label (Class)—A group of similar data inputs in a dataset that your model is trained to recognize. A label references the output name you want your model to predict. For example, the food classifier, which is trained from a multi-label dataset, contains labels like chocolate cake, pasta, carrots, and so on. - Example—A single occurrence or piece of data. For Einstein Vision, an example is an image. For Einstein Language, an example is a text string. Because the learning is supervised, each example is given a label. - Training—The process by which a model is created. A dataset is trained, and the result is a new model. - Model—A machine-learning construct used to solve a classification problem. When you send in images or text, the model returns the prediction. - Prediction—The results that the model returns as to how closely the input matches data in the dataset. - Model metrics—Provides a picture of model accuracy and how well the model will perform. Metrics include elements such as the f1 score, confusion matrix, accuracy of the test data, and accuracy of the training data. ![VisionTerms](https://developer.salesforce.com/media/vision_terms_graphic.png) ## Resources - Blog series: [Machine Learning is Fun!](https://medium.com/@ageitgey/machine-learning-is-fun-80ea3ec3c471) # Dataset and Model Best Practices ## Einstein Language - A dataset can have up to 500 labels, but we recommend a maximum of 100 labels for better model accuracy. - If you have a dataset that contains a lot of classes, increase the number of examples per label. - We recommend that an Einstein Intent or Einstein Sentiment dataset contain a maximum of 100 labels. If you need more than 100 labels, consider hierarchical classification. - We recommend less than 150 words for the length of the intent or sentiment string. This guideline applies to both a language dataset example and a string sent into a model for prediction. - During the training process, special text formatting, like emojis, words in all uppercase , and punctuation aren’t included. For example, if you add a text example containing a smiley emoji to a dataset, the emoji isn’t considered during training. Only the text is used. - When you send in text for prediction, the model doesn’t consider special text formatting and punctuation. For example, when you send the string “We had a great time! :)” to the model, the model returns a prediction for the string “We had a great time”. - Batch predictions aren’t supported. When you send text in for a prediction, you make a single API call to the `/intent` endpoint or the `/sentiment` endpoint. # Dataset Training and Retraining The training and retraining processes both create a model, but they are slightly different. There are different cases when you want to use train or retrain, and this content helps you understand when and why to use each method. ## Training Training is the process by which a model is created from your data (a dataset). In the training process, algorithms are combined with your data to create the object that returns predictions. Each time you call `/train` to train a dataset, it creates another model with a different model ID. Creating a separate model each time you train a dataset is helpful because you can compare them and then test and refine your models. For example, you train dataset A, create a model, and find that the model doesn’t perform to the level you expect. So you add some data to the dataset and train that dataset to create a new model. You can still access the previous model to compare the accuracy and other metrics as you iterate through the process. Here are some considerations to keep in mind when you train a dataset. - When you call `/train`, it creates a new model with a new model ID. - If you decide to use the new model, you must update the model ID in existing code or tools that reference the model ID. - Creating multiple models enables you to compare models as you change your dataset. ![TrainingGraphic](https://developer.salesforce.com/media/training_small.png) ## Retraining Retraining works differently. When you call `/retrain` to train a dataset, it references the dataset used to create an existing model. This method overwrites the old model with the new model, but it retains the previous model ID. For example, you train a dataset, create a model, and iterate through the process. You have an accurate model that you use in your apps. Due to changes in your business, you need to include more data in your model. Here are some considerations to keep in mind when you retrain a dataset. - Retraining overwrites the existing model, but maintains the model ID. - If you retrain using bad or incorrect data, the model is corrupted. You can’t recover the older version. - Because retraining keeps the original model ID, you don’t need to update references to the model ID. Your existing code and tools use the new model. - You must explicitly retrain a dataset when you want to update model. Retraining doesn’t occur automatically. ![RetrainingGraphic](https://developer.salesforce.com/media/retraining_small.png) # Datasets and Models A dataset contains the source text data. A model is created after a dataset is trained. The model is the construct that returns predictions. Here are some key points about datasets and models: - A dataset is the structure that contains your data, whether that data. - The training process uses the dataset to create a model. You train a dataset multiple times to create multiple models. So a single dataset can create many models. - The relationship between a dataset and a model is complete after the model is created. After a model is created, the model doesn’t reference the source dataset again unless you retrain the dataset. - You use APIs to edit a dataset. There’s no visual way to edit a dataset. If you have new data that you want to include in a model, you call the API to create a new dataset. ## Data Collection Tips - For Einstein Language, target 200–500 examples per dataset label. ## Training vs. Retraining a Dataset See [Dataset Training and Retraining](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/dataset-training-and-retraining.md) for more information on when to train a dataset vs. retrain a dataset. ## Training Data and Test Data When you train or retrain a dataset, the training process sets aside some of the training data to test the model for accuracy. - Training data—Examples used by the training process to create the model. - Test data—Examples set aside by the training process to test the model accuracy. - Einstein Language uses 80% of the data to create the model, and 20% is used to test the model’s accuracy. You can change this ratio, also called the split, by using the `trainSplitRatio` parameter. You specify the amount of training and test data, but the actual data that the training process holds out for testing is randomly selected. This means you can see differences in models and model metrics even when those models are created from the same dataset. ## Resources - [Train a Dataset (Language)](/docs/analytics/einstein-vision-language/references) - [Retrain a Dataset (Language)](/docs/analytics/einstein-vision-language/references) # Deep Learning Development Cycle You might already be familiar with the software development life cycle (SDLC). The high-level phases of the SDLC are plan, design, code, test, and deploy. The deep learning life cycle (DLLC) has many similarities to the SDLC, but also a few key differences. If you know how the two differ, you can be prepared and plan better for your deep learning implementation. At a high level, the DLLC phases are identify the desired outcome, gather and prepare data, create the model, and test the model. Here are some key principles of the DLLC. **Data is at the center of deep learning.** Data is the means by which a model is created. So your model is only as good as the data on which it’s based. After you define your use case, or the problem that you’re solving, you must determine if you have enough data. If you don’t have enough good-quality data, the first problem to solve is how to get the data required for your implementation. For example, you want to create a model that analyzes phrases from chat transcripts to identify what a user is trying to accomplish. To create such a model, you need existing chat data. If you implemented chatbot functionality a month ago, you might not yet have enough data to build an accurate model. If you have six months of transcripts from an active chatbot, then you might have enough data. **A lot of time is spent at the beginning gathering data.** This point is a corollary to the previous principle. Even if you know that you have enough data available to create a model, chances are that data requires some processing. If you’re working with text data, you might need to parse phrases out of a transcript. It’s likely that text data needs to be formatted in the way that the API expects it. If you’re working with image data for object detection, you need bounding box data for the items in each image. At the start of your project, you must also define the categories or labels in which your data falls. In addition to gathering “correct” data examples to make predictions for the types of data you expect, you also want to gather “incorrect” data to create a negative label. **The deep learning cycle is iterative.** The model creation process involves frequent loops of model creation and testing. The overall process looks like these steps. 1. Use initial data to create the dataset and model. 2. Test the model. 3. Refine the dataset based on the model test results. 4. Create a new test model, and compare to the previous models. 5. When you’re satisfied with the results, create a final production model from the most recent dataset. You repeat steps 2–4 as often as necessary until you end up with an accurate model. By using Einstein Language, you can focus on the data and quickly get a prototype model up and running. You can use these APIs to get model metrics, test the accuracy, and quickly iterate. **The deep learning cycle is iterative, even after a model is in production.** After a model is in production, you continue to find ways to improve it. For example, you might find that certain data is consistently misclassified, perhaps data that you didn’t expect to be sent for classification. In this case, you might want to add a label and the data to the dataset and retrain it. Your business might change and require adding a new label and data to the dataset and model. Expect that your models continue to change even after they’re in production. It’s important to expect these changes so that you can allocate resources and time to maintaining the model. ![AgileAISprints](https://developer.salesforce.com/media/agile_ai_sprints_ran_lee.png) # Detect Text and Entities in Business Cards To return text for a business card, specify a modelId of `OCRModel` and `task` parameter of `contact`. In addition to the text, the model returns the entity type for each text element that it detects. ![BusinessCardOriginal](https://developer.salesforce.com/media/business_card_original.jpeg) The cURL call looks like this. ```cURL curl -X POST -H "Authorization: Bearer " -F sampleLocation="https://www.asianbusinesscards.com/wp-content/uploads/2019/03/chinese-business-card-translation-samples-walmart-890-eng.jpg" -F task="contact" -F modelId="OCRModel" https://api.einstein.ai/v2/vision/ocr ``` ![BusinessCardOCRWithCallouts](https://developer.salesforce.com/media/business_card_ocr.png) The response looks like this JSON. ```json "tag": "OTHER" } }, { "probability": 0.9989197, "label": "Director Operations and Logistics", "boundingBox": { "minX": 53, "minY": 105, "maxX": 466, "maxY": 134 }, "attributes": { "tag": "OTHER", "pageNumber": "1" } }, { "probability": 0.9978701, "label": "Randy E. Cornwell", "boundingBox": { "minX": 53, "minY": 73, "maxX": 297, "maxY": 103 }, "attributes": { "tag": "PERSON", "pageNumber": "1" } }, { "probability": 0.99883646, "label": "S-1050-06 Headquarters Building", "boundingBox": { "minX": 55, "minY": 384, "maxX": 467, "maxY": 413 }, "attributes": { "tag": "OTHER", "pageNumber": "1" } }, { "probability": 0.9981871, "label": "Houston TX 77042 USA 2881 CityWest Blvd.", "boundingBox": { "minX": 53, "minY": 416, "maxX": 350, "maxY": 472 }, "attributes": { "tag": "ADDRESS", "pageNumber": "1" } }, { "probability": 0.9984518, "label": "284.575.2884", "boundingBox": { "minX": 547, "minY": 384, "maxX": 820, "maxY": 412 }, "attributes": { "tag": "PHONE", "pageNumber": "1" } }, { "probability": 0.9984518, "label": "phone ", "boundingBox": { "minX": 547, "minY": 384, "maxX": 820, "maxY": 412 }, "attributes": { "tag": "OTHER", "pageNumber": "1" } }, { "probability": 0.9973475, "label": "281.889.0997", "boundingBox": { "minX": 541, "minY": 415, "maxX": 825, "maxY": 439 }, "attributes": { "tag": "MOBILE_PHONE", "pageNumber": "1" } }, { "probability": 0.9973475, "label": "mobile ", "boundingBox": { "minX": 541, "minY": 415, "maxX": 825, "maxY": 439 }, "attributes": { "tag": "OTHER", "pageNumber": "1" } }, { "probability": 0.89205074, "label": "re.comwell@phillips86.com", "boundingBox": { "minX": 478, "minY": 444, "maxX": 782, "maxY": 476 }, "attributes": { "tag": "EMAIL", "pageNumber": "1" } }, { "probability": 0.992919, "label": "Phillips", "boundingBox": { "minX": 692, "minY": 79, "maxX": 780, "maxY": 100 }, "attributes": { "tag": "OTHER", "pageNumber": "1" } }, { "probability": 0.90038645, "label": "66", "boundingBox": { "minX": 693, "minY": 113, "maxX": 777, "maxY": 162 }, "attributes": { "tag": "OTHER", "pageNumber": "1" } }, { "probability": 0.9908977, "label": "Phillips66.com", "boundingBox": { "minX": 53, "minY": 198, "maxX": 238, "maxY": 226 }, "attributes": { "tag": "WEBSITE", "pageNumber": "1" } }, { "probability": 0.89205074, "label": "phillips86", "boundingBox": { "minX": 630, "minY": 444, "maxX": 782, "maxY": 476 }, "attributes": { "tag": "ORG", "pageNumber": "1" } } ], "object": "predictresponse" } ``` # Detect Text and Tables Some images contain tabular data. To return the table data for each text element, specify the `tabulatev2` model and a `task` parameter value of `table`. ![TableOriginal](https://developer.salesforce.com/media/table_original.png) The cURL call looks like this. ```cURL curl -X POST -H "Authorization: Bearer " -F sampleContent="@/filepath/table.png" -F task="table" -F modelId="tabulatev2" https://api.einstein.ai/v2/vision/ocr ``` The response looks like this JSON. ```json { "task": "table", "probabilities": [ { "probability": 0.9952141, "label": "City", "boundingBox": { "minX": 158, "minY": 50, "maxX": 214, "maxY": 82 }, "attributes": { "cellLocation": { "rowIndex": 1, "colIndex": 1 }, "pageNumber": "1" } }, { "probability": 0.99889404, "label": "Number of Events", "boundingBox": { "minX": 405, "minY": 50, "maxX": 659, "maxY": 75 }, "attributes": { "cellLocation": { "rowIndex": 1, "colIndex": 2 }, "pageNumber": "1" } }, { "probability": 0.9988783, "label": "Number of Stations", "boundingBox": { "minX": 773, "minY": 51, "maxX": 1048, "maxY": 75 }, "attributes": { "cellLocation": { "rowIndex": 1, "colIndex": 3 }, "pageNumber": "1" } }, { "probability": 0.99629766, "label": "San Francisco", "boundingBox": { "minX": 91, "minY": 115, "maxX": 282, "maxY": 140 }, "attributes": { "cellLocation": { "rowIndex": 2, "colIndex": 1 }, "pageNumber": "1" } }, { "probability": 0.9942451, "label": "100", "boundingBox": { "minX": 511, "minY": 116, "maxX": 557, "maxY": 140 }, "attributes": { "cellLocation": { "rowIndex": 2, "colIndex": 2 }, "pageNumber": "1" } }, { "probability": 0.97811675, "label": "21", "boundingBox": { "minX": 894, "minY": 116, "maxX": 924, "maxY": 139 }, "attributes": { "cellLocation": { "rowIndex": 2, "colIndex": 3 }, "pageNumber": "1" } }, { "probability": 0.9973557, "label": "San Diego", "boundingBox": { "minX": 117, "minY": 180, "maxX": 256, "maxY": 213 }, "attributes": { "cellLocation": { "rowIndex": 3, "colIndex": 1 }, "pageNumber": "1" } }, { "probability": 0.99561936, "label": "123", "boundingBox": { "minX": 510, "minY": 182, "maxX": 558, "maxY": 206 }, "attributes": { "cellLocation": { "rowIndex": 3, "colIndex": 2 }, "pageNumber": "1" } }, { "probability": 0.9816425, "label": "18", "boundingBox": { "minX": 896, "minY": 181, "maxX": 926, "maxY": 206 }, "attributes": { "cellLocation": { "rowIndex": 3, "colIndex": 3 }, "pageNumber": "1" } }, { "probability": 0.9973557, "label": "San Jose", "boundingBox": { "minX": 125, "minY": 246, "maxX": 248, "maxY": 271 }, "attributes": { "cellLocation": { "rowIndex": 4, "colIndex": 1 }, "pageNumber": "1" } }, { "probability": 0.9696964, "label": "53", "boundingBox": { "minX": 517, "minY": 247, "maxX": 549, "maxY": 270 }, "attributes": { "cellLocation": { "rowIndex": 4, "colIndex": 2 }, "pageNumber": "1" } }, { "probability": 0.9926605, "label": "189", "boundingBox": { "minX": 888, "minY": 247, "maxX": 935, "maxY": 271 }, "attributes": { "cellLocation": { "rowIndex": 4, "colIndex": 3 }, "pageNumber": "1" } } ], "object": "predictresponse" } ``` The above JSON describes Einstein's detection of the table elements, and is depicted below: ![TableWithOCRCallouts](https://developer.salesforce.com/media/table_ocr_with_caps.png) ## Multiple Tables on a Page When an image contains more than one table, Einstein returns tabular data for each table in the image. The following image depicts how the tables and fields are denoted in the JSON. (The JSON is not provided). With this example, three tables are identified: One 3x2, one 2x5, and one 2x2. The values in the second row of the 2x2 table cells will be identified separately, resulting in a 2x6 JSON table definition. ![MultiTableExample](https://developer.salesforce.com/media/multi-table-example.png) **note:** We currently support only the content confined within the individual tables. Free text in the image which lies outside of a table border is not processed. # Detect Text in Standard Forms Einstein OCR (optical character recognition) can now extract text from a variety of common forms. Einstein OCR supports these standard forms. - Driver's license, only United States, including Washington, DC - Form 1040, only 2019 - Form W-2 - Passport, only Australia, Canada, United Kingdom, and United States - Pay stub, generated by ADP or Workday - Permanent Resident Card, only United States including Washington, DC When you call the API, you send in the form as an image or PDF, and specify the `tabulatev2` modelId. The JSON response contains key-value pairs for each field in the form. ![JaneDoeDriversLicenseWithCallouts](https://developer.salesforce.com/media/jane_doe_dl_with_callouts.png) 1. The **key** is the label of the form field detected by Einstein OCR. If you fill out the form, the key is the text that you read to see what data to put in that field. 2. The **value** is the data of the actual form field. The model infers the document layout and word relationships, then extracts the entity and value for each field in the form. For example, in a driver's license, Einstein links together the issue date of the license (09/13/2020), how the issue date is referred to on the license (4a ISS), and the entity type (issue_date), which is a consistent field name across all variations of driver’s licenses. In addition, when the model detects a compound field (such as an address), it normalizes the text of that field into its constituent parts. Here are some examples of key-value pairs from a form. | Form | Form Key | Form Value | Normalized Text | | ---------------- | ------------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------ | | Form 1040 | `a Employee's social security number` | 123-45-6789 | n/a | | Driver’s license | `3 DOB` | 09/13/1999 | n/a | | Form W-2 | `Employee Address` | 123 Sample Street Phoenix, AZ, 01234 | `{\\”Street\\”:\\”123 Sample Street\\”,\\”City\\”Phoenix\\”,\\”State\\”:”\\AZ\\”,”\\ZipCode\\”:01234}` | When we detect a form field as an address, we normalize that text and return fields populated with street, city, state, postal code, and country. When an address has multiple lines (as in the case of _P.O. Box_, or _Apartment_), **normalizedText** returns the original text. if there’s a line separator in the text field, it will be present in the **normalizedText** field. **note:** Currently only U.S. addresses are normalized. See the [Entity Normalization](https://developer.salesforce.com/docs/analytics/einstein-vision-language/Detect-Entities-in-Text/entity-normalization.md) for more details on how detected text is normalized. # Driver's License This table lists the entities returned by the OCR model for a driver's license. Supports licenses only from the United States, including Washington, DC. | Entity name | Additional info | Possible form keys | | ------------------------ | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `address` | | 8
8 ADDRESS | | `audit` | Louisiana driver’s license only. | AUDIT | | `class` | | 9 Class | | `date_of_birth` | | 3 DOB
Date of Birth
D.O.B.
DOB | | `document_discriminator` | | 5 DD
DD | | `donor` | | ORGAN DONOR
DONOR | | `duplicate_number` | Alaska driver’s license only. | DUPLICATE NUM | | `dups` | Pennsylvania driver’s license only. | | | `endorsements` | | 9a End
9a END
9a ENDORSEMENTS
End
END | | `expiration_date` | | Expires
4b EXP | | `eyes` | | 18 Eyes
18 EYES
EYES | | `first` | Oregon driver’s license only. | FIRST | | `first_name` | | 2
FN | | `gender` | | 15 Sex
15 SEX
Sex
SEX | | `hair` | | 19 Hair
19 HAIR
Hair
HAIR | | `height` | | 16 HGT
16 Hgt
Height
HGT | | `issue_date` | | 4a ISS
Issued
ISS | | `last_name` | | 1
LN | | `name` | Full name. Returned only when the first name and last name are on the same line. | | | `number` | | 4d DLN
4d DL NO.
DL
Number | | `office` | Louisiana driver’s license only. | | | `parish` | Louisiana driver’s license only. | | | `previous_type` | Colorado driver’s license only. | | | `race` | North Carolina driver’s license only. | | | `replaced` | Florida driver’s license only. | | | `restrictions` | | 12 Rest
12 REST:
12 REST
12 RESTRICTIONS
12 Restrictions
RESTRICTIONS
Restrictions
RSTR | | `social_security_number` | Louisiana driver’s license only. | | | `state` | | | | `type` | Illinois driver’s license only. | | | `weight` | | 17 Wgt
17 WGT
WGT | # Einstein Intent Quick Start This quick start shows you how to use the Einstein Intent API to create a model to route support cases. You can then use this model to analyze text and infer what the user wants to accomplish. ## Prerequisites - **Find your key file**—If you already created an account, locate the `einstein_platform.pem` file that you downloaded as part of that process. This file contains your private key. - **Find your key file**—If you already created an account, locate the `einstein_platform.pem` file that you downloaded as part of that process. This file contains your private key. - **Install cURL**—We use the cURL command line tool throughout the following steps. This tool is installed by default on Linux and OSX. If you don’t already have it installed, download it from [https://curl.haxx.se/download.html](https://curl.haxx.se/download.html) - **Get a Token**—The Einstein Vision and Language APIs use OAuth 2.0 JWT bearer token flow for authorization. Use the [token page](https://api.einstein.ai/web/token) to upload your key file and generate a JWT token. For step-by-step instructions, see [Set Up Authorization](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/set-up-auth.md). ## Step 1: Define Your Classes and Gather Data In this step, you define the labels that you want the model to output when text is sent into the model for prediction. Then you gather text data for each of those labels, and that text is used to create a model. This is typically the most time-consuming part of of the process. To make it easier for you to go through these steps, we provide a case routing [.csv file](http://einstein.ai/text/case_routing_intent.csv) that you can use. The labels in the case routing dataset define the intent behind the text. The intent can then be used to route that case to the right department. Those labels are: - `Billing` - `Order Change` - `Password Help` - `Sales Opportunity` - `Shipping Info` ## Step 2: Create the Dataset In this step, you use the data you gathered to create a dataset. In the following command, replace `` with your JWT token and run the command. This command: - Creates a dataset called `case_routing_intent` from the specified .csv file by accessing the file via a URL - Creates five labels as specified in the .csv file - Creates 150 examples ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "data=case_routing_intent.csv" -F "type=text-intent" https://api.einstein.ai/v2/language/datasets/upload ``` This call is asynchronous, so the response looks like this JSON. ```json { "id": 1004804, "name": "case_routing_intent.csv", "createdAt": "2019-06-22T19:31:58.000+0000.", "updatedAt": "2019-06-22T19:31:58.000+0000", "labelSummary": { "labels": [] }, "totalExamples": 0, "available": false, "statusMsg": "UPLOADING", "type": "text-intent", "language": "en_US", "numOfDuplicates": 0, "object": "dataset" } ``` To verify that the data has been loaded, make a call to get the dataset. Replace `` with your JWT token and `` with ID of the dataset you created. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/datasets/ ``` The results look something like this JSON. You know the dataset is ready when `available` is `true` and `statusMsg` is `SUCCEEDED`. ```json { "id": 1004804, "name": "case_routing_intent.csv", "createdAt": "2019-06-22T19:31:58.000+0000", "updatedAt": "2019-06-22T19:31:59.000+0000", "labelSummary": { "labels": [ { "id": 23649, "datasetId": 1004804, "name": "Order Change", "numExamples": 26 }, { "id": 23650, "datasetId": 1004804, "name": "Sales Opportunity", "numExamples": 44 }, { "id": 23651, "datasetId": 1004804, "name": "Billing", "numExamples": 24 }, { "id": 23652, "datasetId": 1004804, "name": "Shipping Info", "numExamples": 30 }, { "id": 23653, "datasetId": 1004804, "name": "Password Help", "numExamples": 26 } ] }, "totalExamples": 150, "totalLabels": 5, "available": true, "statusMsg": "SUCCEEDED", "type": "text-intent", "language": "en_US", "numOfDuplicates": 0, "object": "dataset" } ``` ## Step 3: Train the Dataset to Create the Model Use this cURL command to train the dataset and create a model. Replace `` with your JWT token and `` with ID of the dataset you created. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Case Routing Model" -F "datasetId=" https://api.einstein.ai/v2/language/train ``` The response looks like this JSON. ```json { "datasetId": 1004804, "datasetVersionId": 0, "name": "Case Routing Model", "status": "QUEUED", "progress": 0, "createdAt": "2017-06-22T19:39:38.000+0000", "updatedAt": "2017-06-22T19:39:38.000+0000", "modelId": "5SXGNLCCOFGTMNMQYEOTAGBPVU", "learningRate": 0, "epochs": 0, "queuePosition": 1, "modelType": "text-intent", "algorithm": "multilingual-intent", "language": "en_US", "object": "training", "trainParams": null, "trainStats": null } ``` Use the `modelId` to make this call and get the training status. Replace `` with your JWT token and `MODEL_ID` with the ID of the model you created. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/train/ ``` The training status response looks like this JSON. The `status` of `SUCCEEDED` means that the model is ready for predictions. ```json { "datasetId": 1004804, "datasetVersionId": 2473, "name": "Case Routing Model", "status": "SUCCEEDED", "progress": 1, "createdAt": "2019-06-22T19:39:38.000+0000", "updatedAt": "2019-06-22T19:43:39.000+0000", "modelId": "5SXGNLCCOFGTMNMQYEOTAGBPVU", "learningRate": 0, "epochs": 300, "modelType": "text-intent", "algorithm": "multilingual-intent", "language": "en_US", "object": "training", "trainParams": null, "trainStats": { "labels": 5, "examples": 150, "averageF1": 0.9492063492063492, "totalTime": "00:03:54:159", "transforms": null, "trainingTime": "00:03:53:150", "earlyStopping": true, "lastEpochDone": 267, "modelSaveTime": "00:00:01:561", "testSplitSize": 28, "trainSplitSize": 122, "datasetLoadTime": "00:00:01:008", "preProcessStats": null, "postProcessStats": null } } ``` ## Step 4: Send Text in for Prediction Now your model is ready to go! To test it out, send some text in for prediction. This cURL call takes the `modelId` of the model from which you want to return a prediction and the text string to analyze. Replace `` with your JWT token and `` with the ID of your model. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "modelId=" -F "document=how is my package being shipped?" https://api.einstein.ai/v2/language/intent ``` The response looks like this JSON. The model predicts that the text indicates that the user has a comment or question about shipping, so the model returns `Shipping Info` as the top probability. Your app can then use this information to route the case to the right department or agent. ```json { "probabilities": [ { "label": "Shipping Info", "probability": 0.82365495 }, { "label": "Sales Opportunity", "probability": 0.12523715 }, { "label": "Billing", "probability": 0.0487557 }, { "label": "Order Change", "probability": 0.0021365683 }, { "label": "Password Help", "probability": 0.0002156619 } ], "object": "predictresponse" } ``` # Einstein OCR Model Card The model analyzed in this card returns text, probabilities, and other information identified in an image or PDF. Get information about training data, use cases, features, and other considerations for the optical character recognition (OCR) model. ## Model Details ### Person or Organization Salesforce AI Research ### Model Date and Version - February 2022 - The model doesn’t have numbered versioning. When a fix is sent out, it's deployed everywhere so customers are always running on the latest version. - Minor changes can occur throughout the release. - Major changes can occur and are communicated via release notes. ### Model Type Vision model for character recognition. Einstein OCR has three modules. The first module detects the text in an image. The second module identifies characters in the text that was identified in the first module, and converts it character by character to a string. The third module uses this computed text to tag the text with an entity such as person, phone number, or address. The modules use a convolutional neural network (CNN) plus a combination of a CNN and long short-term memory (LSTM) network. ### License Einstein OCR is available to Salesforce customers that have any SKU for Einstein Vision and Language. ### Questions or Comments Send questions or comments about the model to vision-team@salesforce.com. For more info, see [Einstein OCR documentation](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/einstein-ocr.md). ## Intended Use The Einstein OCR model detects alphanumeric text and common special characters (such as ’#’ and ‘&’) in an image. ### Primary Intended Users The Einstein OCR model is designed for enterprise customers. Users can access the model via REST API or embed the API in a Salesforce application. ### Primary Intended Uses - **OCR text:** Identifies typed and handwritten text in a document or image and returns the relative position of that text. Use cases include: - VIN number detection - serial number detection - **OCR entity:** Identifies text in an image and returns the relative position and the associated entity. To build on OCR text, it identifies text and then identifies the entity that the text represents. Use cases include extracting contact information from business cards. - **OCR table:** Identifies text in an image and returns the relative position and associated cell number. To build on OCR text, it identifies tables in the text and maps entries to a specific table cell. Use cases include digitizing tables such as price sheets. ### Out-of-Scope Uses - The Einstein OCR model currently only supports English. See the Note section for details on supported characters. - The model doesn't support checkboxes, or circled answers. - The model is optimized for short-form prediction rather than the identification of characters, words, or entities in long-form sentences or paragraphs. - Certain use cases are prohibited under the [Salesforce Acceptable Use Policy](https://www.salesforce.com/content/dam/web/en_us/www/documents/legal/Agreements/policies/ExternalFacing_Services_Policy.pdf). - A user can't submit data prohibited by the [Security, Privacy and Architecture](https://www.salesforce.com/content/dam/web/en_us/www/documents/legal/misc/einstein-platform-security-privacy-and-architecture.pdf) documentation, Sensitive Data section. ## Limitations The quality of a user's input can affect the accuracy of the results. These factors can affect text recognition accuracy. - Low quality images such as those downloaded from unknown sources or poorly scanned - Low light - Low resolution - High skew or perspective - Varying density of words in the document - Length of strings - Excessively cursive or handwritten text that's illegible - Font of the typed text These factors can affect entity recognition accuracy. - Addresses with varied formats that differ between countries. For example, apartment numbers, postal codes, and so on - Birth dates in different formats - Names in different cultures - Various genders ## Metrics **Precision and recall** are used to evaluate where the text is within an image (the bounding box). **Word-level accuracy** is used to evaluate character recognition in identified text. The entity **F1** score is used to discern the accuracy of entity tags identified for text. The overall model is also evaluated using an **F1 score**. The score calculation is based on the precision and recall and the word-level accuracy mentioned above. ## Training and Evaluation Data The training dataset is composed of data from these open-source libraries: - Born-Digital Images, found on [https://rrc.cvc.uab.es](https://rrc.cvc.uab.es), in the Challenges picklist - Focused Scene Text, found on [https://rrc.cvc.uab.es](https://rrc.cvc.uab.es), in the Challenges picklist - MLT, found on [https://rrc.cvc.uab.es](https://rrc.cvc.uab.es), in the Challenges picklist - COCO-Text, found on [https://rrc.cvc.uab.es](https://rrc.cvc.uab.es), in the Challenges picklist - IIIT-HWS, found on https://github.com/kris314/hwnet/tree/master/iiit-hws (1) - The UNIPEN on-line handwritten samples collection, found on http://www.unipen.org/products.html (2) In addition to the open-source libraries, synthetic datasets were used. Synthetic datasets were created by picking a scene or plain white background and then imposing randomly selected words onto the background and changing certain attributes such as font, character size, or skew. The entity recognition capabilities use a combination of in-house named-entity recognition (NER) models, the Stanford business card dataset, and an in-house business card dataset. The data used to evaluate the model was gathered from the sources listed above. The evaluation and training data sets are different subsets of the source data that don’t overlap. ## Quantitative Analyses The results reported here are from a publicly available challenge dataset, [ICDAR Focused scene text](https://rrc.cvc.uab.es/?ch=2&com=introduction). This dataset consists of real-world scenes such as storefronts and street signs, with images primarily focused on text content. This scenario is typical for text reading applications where the user explicitly directs the camera’s focus on the text content of interest. The challenge in this dataset arises from complex text orientation and complex backgrounds. Complex text orientation necessitates estimating skew in addition to the location and content of the text. Complex backgrounds can reduce contrast of the text and introduce false positives in the form of random patterns that appear like text. The ICDAR dataset was designed to encourage the research community to develop solutions for text reading in real-world images, and the challenge allowed for the OCR system to limit the words to a fixed vocabulary (as with the State of the Art (SoTA) benchmark reported below). Such a limitation can be helpful in correcting OCR errors because the words to be detected in the test set are present in the fixed vocabulary. And the search space for a word in a test image decreases from potentially infinite (all possible combinations of alphabets and numbers) to a small number of words in the specified vocabulary. The SoTA models reported here use a general vocabulary set to aid in text prediction, and are trained on this dataset. But our results don’t use a vocabulary because the models support OCR applications that can’t be captured by a vocabulary, such as vehicle identification number (VIN) or serial number scanning. The models aren't trained on this dataset exclusively. A generic model is used that’s more robust and addresses a wider range of challenges. The F1 score diverges from the SoTA results by just 6.17%. **Evaluation Metric:** F1 score **Salesforce Research Model ID:** `scene` | | **Salesforce Research** | **SoTA Research Benchmark\*** | | -------------------- | ----------------------- | ----------------------------- | | **Fixed Vocabulary** | No | Yes | | **Usage** | Real world | Fine tuned for the dataset | | **F1 Score** | 79.61 | 85.78 | \*[SoTA Research Benchmark](https://rrc.cvc.uab.es/?ch=2&com=evaluation&task=4&f=3&e=1) ## Ethical Considerations The text is processed into a string character by character. Because dictionaries aren’t referenced, underlying bias is not an issue. But the supported characters are limited to the Roman alphabet, which could be advantageous to English-language individuals, companies, and organizations. Certain entities are identified after the text is converted character by character to a string. So the identification of certain entities may be challenging for the model. For example, first or last names from certain regional, ethnic, or religious groups sometimes are recognized with higher accuracy. This issue can include non-binary gender or varied birth date formats. ## Note - Only text that contains characters found in the supported character list is identified. Languages with characters other than those in the supported character list aren’t supported. An increase in the supported characters and languages is planned. - Only entities in the supported entity list are identified in this version of the model. - Only phone numbers displayed in the format of the supported phone number formats are identified in this version of the model. - The Detect Text API provides for more than one prediction model. Each has a unique modelId specifier (for example, _OCRModel_ and _tabulatev2_). For best accuracy, use the modelId and task combinations we recommend. ### Supported Characters These special characters are supported. `` !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~£≈ `` ### Supported Entities These entities are supported. - `ADDRESS` - `EMAIL` - `FAX` - `HOME_PHONE` - `MOBILE_PHONE` - `OFFICE_PHONE` - `ORG` - `PERSON` - `WEBSITE` ### Phone Number Formats Phone numbers with these country-specific formats are supported. - `CA` - `CN` - `DE` - `ES` - `FR` - `GB` - `IN` - `IT` - `MX` - `JP` - `PT` - `US` ### Service Numbers We can also identify service and emergency numbers (such as 911) in the locales listed below. Our support for service numbers is limited to these specific locales. | **Locale** | **Locale Code** | **Service Number** | | ---------------------------- | --------------- | ------------------ | | Chinese, Simplified (China) | zh_cn | 110, 119, 120 | | Chinese, Traditional (China) | zh_tw | 110,119,120 | | Danish (Denmark) | da | 112 | | Dutch (Netherlands) | nl_nl | 112 | | English (U.S.A.) | en_us | 911 | | English (U.K) | en_gb | 999, 112 | | French (France) | fr | 15, 17, 18, 112 | | German (Germany) | de | 110, 112 | | Italian (Italy) | it | 112 | | Japanese (Japan) | ja | 110, 119 | | Korean (Korea) | ko | 112, 119 | | Portuguese (Brazil) | pt_br | 190, 192, 193 | | Portuguese (Portugal) | pt_pt | 112 | | Russia (Russian) | ru | 101,102, 103, 112 | | Spanish (Spain) | es | 112 | | Swedish (Sweden) | sv | 112 | (1) Praveen Krishnan and C.V. Jawahar, Generating Synthetic Data for Text Recognition, arXiv preprint arXiv:1608.04224, 2016. (2) The database is provided by the [Unipen Foundation](http://www.unipen.org/images/178_unipen-logo.jpg). Einstein Vision and Language API # Einstein Vision and Language API * * * **Important:** Salesforce is retiring Einstein Vision and Language, and your subscription will not be renewable after May 1, 2024. You can continue using Einstein Vision and Language through your Order Term. To avoid interruptions, transition any Salesforce org using Einstein Vision and Language to another solution before your subscriptions expire. We will be communicating additional details to you soon. If you have questions, contact your Account Executive about your subscriptions. * * * You access the Einstein Vision and Einstein Language services via these standard REST API calls. Use the APIs to programmatically work with datasets, labels, examples, models, and predictions. **Einstein Vision and Language Are HIPAA Compliant** Einstein Vision and Language are Health Insurance Portability and Accountability Act (HIPAA) compliant. Einstein Vision and Language meet the strict HIPAA standards for processing customers' protected health information (PHI). HIPAA compliance positions Einstein Vision and Language services to provide capabilities that can be used in the highly regulated healthcare sector. ### API Error Codes and Messages If an API call is unsuccessful, it returns an HTTP error code. If the error is known, you receive a message in the response body. Known errors are returned in the response body in this format. ```json { "message": "Invalid authentication scheme" } ``` #### All | HTTP Code | HTTP Message | API Message | Resource | Possible Causes | | --- | --- | --- | --- | --- | | 401 | Unauthorized | Invalid access token | Any | The access token is expired. | | 401 | Unauthorized | Invalid authentication scheme | Any | An `Authorization` header was provided, but the token isn't properly formatted. | | 5XX | Internal server error; Service unavailable | None | Any | Our systems encountered and logged an unexpected error. Please contact us if you continue to see the error. | #### Datasets Error codes that can occur when you access datasets, labels, or examples. | HTTP Code | HTTP Message | API Message | Resource | Possible Causes | | --- | --- | --- | --- | --- | | 400 | Bad Request | None | Any dataset, label, or example resources. | The request couldn't be fulfilled because the HTTP request was malformed, the Content Type was incorrect, there were missing parameters, or a parameter was provided with an invalid value. | | 400 | Bad Request | The `name` parameter is required to create a dataset. | `POST /language/datasets` | The `name` parameter was passed in, but no value was provided. | | 400 | Bad Request | Uploading a dataset requires either the `data` field. | `POST /language/datasets/upload` | The path to the local .zip file or the URL to the .zip file in the cloud wasn't specified. | | 400 | Bad Request | The `data` parameter cannot be duplicated. | `POST /language/datasets/upload` | The `data` parameter cannot be duplicated. | | 400 | Bad Request | The dataset is not yet available for update, try again once the dataset is ready. | `PUT /language/datasets//upload` | You're adding examples to a dataset that's currently being created. You must wait for the dataset to become available before you can add examples to it. | | 400 | Bad Request | Failed to download the dataset from the public URL. | `POST /language/datasets/upload`; `POST /language/datasets/upload/sync` | The API can't access the dataset file via the URL provided. When specifying the URL for a .zip file in a cloud drive service like Dropbox, be sure it's a direct file link, not an interactive download page. Example: `https://www.dropbox.com/s/abcdxyz/mountainvsbeach.zip?dl=1`. | | 400 | Bad Request | Supported dataset types: \[image, image-detection, image-multi-label\] | `POST /language/datasets/upload` | The `type` request parameter contains a value that isn't a valid dataset type. | | 400 | Bad Request | Example max size supported is 1024000. | `POST /language/datasets//examples` | The image file being added as an example exceeds the maximum file size of 1 MB. | | 404 | Not Found | None | Any dataset, label, or example resources. | The requested REST resource doesn't exist or you don't have permission to access the resource. | | 404 | Not Found | Unable to find dataset. | `GET /language/datasets/` | You don't have access to the dataset, or the dataset was deleted. | | 404 | Not Found | Unable to find dataset. | `DELETE /language/datasets/` | You don't have access to the dataset, or the dataset was deleted. | | 404 | Bad Request | Duplicate labels are not allowed. | `POST /language/datasets` | The call is trying to create a label with a name that exists in the dataset. Label names must be unique within a dataset. | | 503 | Service Unavailable | Operation timed out! | `GET /language/datasets` | The call has timed out due to large data size. By default, this call returns 25 datasets. If the datasets contain many examples, use the `offset` and `count` parameters to limit and page through the data. | #### Training Error codes that can occur when you train a dataset to create a model or access a model. | HTTP Code | HTTP Message | API Message | Resource | Possible Causes | | --- | --- | --- | --- | --- | | 400 | Bad Request | The `name` parameter is required to train; a valid `datasetId` parameter is required to create an example. | `POST /language/train` | The `name` or `datasetId` parameter was passed in, but no value was provided. | | 400 | Bad Request | The `name` or `datasetId` parameters are required to train. | `POST /language/train` | The `name` or `datasetId` parameter is missing. | | 400 | Bad Request | Invalid id `` | `GET /language/train/` | There's no model with an ID that matches the `modelId` parameter. | | 400 | Bad Request | Invalid id `` | `GET /language/train//lc` | There's no model with an ID that matches the `modelId` parameter. | | 400 | Bad Request | The job has not terminated yet; its current status is RUNNING or QUEUED. | `GET /language/models/` | The model for which you are getting metrics hasn't completed training. | | 404 | Not Found | None | `GET /language/models/` | The `modelId` parameter is missing. | | 405 | Method Not Allowed | None | `GET /language/train/` | The `modelId` parameter is missing. | #### Prediction Error codes that can occur when you make a prediction. | HTTP Code | HTTP Message | API Message | Resource | Possible Causes | | --- | --- | --- | --- | --- | | 400 | Bad Request | None | `POST /language/intent` | The prediction request couldn't be fulfilled because the HTTP request was malformed, the Content Type was incorrect, there were missing parameters, or a parameter was provided with an invalid value. | | 400 | Bad Request | Bad Request: Bad sampleLocation | `POST /language/intent` | The URL passed in the `sampleLocation` parameter is invalid. The URL could be incorrect, contain the wrong file name, or the file may have been moved. | | 400 | Bad Request | The `modelId` parameter is required. | `POST /language/intent` | The `modelId` parameter is missing. | | 400 | Bad Request | File size limit exceeded | `POST /language/intent` | The file you passed in for prediction exceeds the maximum file size limit of 5 MB. | | 400 | Bad Request | Bad Request: Unsupported sample file format | `POST /language/intent` | The file you passed in for prediction isn't one of the supported file types. | | 403 | Forbidden | Forbidden! | `POST /language/intent` | The model specified by the `modelId` parameter doesn't exist; or the `modelId` parameter was passed in but no value was provided. | | 429 | Too Many Requests | You've reached the maximum number of predictions. | `POST /language/intent` | You have exceeded the number of prediction requests for your current plan. Contact your AE to update your plan. See [Rate Limits](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/rate-limits.html). | **Version:** 1.0 Contact: Salesforce Einstein, einstein-ep-appservicefoundation@salesforce.com, https://www.einstein.com Terms of service: https://developer.salesforce.com/files/tos/Developerforce\_TOU\_20101119.pdf ## Servers * `https://api.einstein.ai` Default security: bearerAuth. ## Security * `bearerAuth` (http) Use a bearer access token in the `Authorization` header: `Authorization: Bearer `. Obtain access tokens by calling `POST /v2/oauth2/token`. * HTTP scheme: bearer (OAuth 2.0 access token) * `einsteinOAuth2` (oauth2) Einstein Vision and Language issues access tokens from `POST /v2/oauth2/token`. This API accepts form-encoded OAuth token grant parameters such as `grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer` and `grant_type=refresh_token`. OpenAPI 3.0 does not model the JWT bearer grant exactly, so this scheme documents the token URL and access-token usage. * OAuth flow clientCredentials (tokenUrl: https://api.einstein.ai/v2/oauth2/token) ## Tags ### OAuth Token Generate and delete OAuth tokens with these API calls. ### API Usage Returns prediction usage on a monthly basis for the current calendar month and future months. ### Datasets (Language) Create, get, and delete Einstein Language datasets using these API calls. ### Examples (Language) Create, get, and delete Einstein Language examples using these API calls. ### Training (Language) Train, retrain, and get training status of Einstein Language datasets with these API calls. ### Models (Language) Get Einstein Language models, model metrics, and model learning curves with these API calls. ### Predictions (Language) Resources that return predictions for text input. ### Vision Prediction Get Einstein Language models, model metrics, and model learning curves with these API calls. ## Operations ### GET Get API Usage `GET /v2/apiusage` Operation ID: `getApiUsageV2`. Tags: API Usage. Each `apiusage` object in the response contains plan information for a single calendar month for a single license. If you have a six-month paid plan and you make this call on the first month, the response contains six `apiusage` objects; one for each calendar month in the plan. For more information about plans, see [Rate Limits](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/rate-limits.html). This the cURL request for this call. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/apiusage ``` * If you're using the free tier, the response contains plan information only for the current month. You see plan information only after you make your first prediction. If you call the `/apiusage` resource before you make your first prediction call, the API returns an empty array. * If you're using the paid tier, the response contains plan information for each month in your plan starting with the current month. The `planData` array contains an object for each plan type associated with the calendar month and the license. This code snippet shows the `planData` if the user has two Heroku GOLD plans. ```json "planData": [ { "plan": "GOLD", "amount": 2, "source": "HEROKU" } ] ``` #### Responses * **200** API usage Media type: `application/json`. Schema: `ApiUsageList`. * **500** Failed to get usage ### DELETE Delete a Refresh Token `DELETE /v2/oauth2/tokens/{refreshToken}` Operation ID: `deleteRefreshToken`. Tags: OAuth Token. This call doesn't return a response body. Instead, it returns an HTTP status code `204`. This is the cURL request. ```cURL curl -X DELETE -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/oauth2/tokens/ ``` #### Parameters * `Authorization` (header, string) * `refreshToken` (path, string, required) The refresh token to revoke. _Example_ ``` SOME_REFRESH_TOKEN ``` #### Responses * **204** Token is deleted successfully * **400** Bad request ### POST Generate an OAuth Token `POST /v2/oauth2/token` Operation ID: `generateToken`. Tags: OAuth Token. Returns an OAuth access token. You must pass a valid access token in the header of each API call. For information about how to create an access token, see [Generate an OAuth Token Using Your Key](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/generate-an-oauth-token-using-your-key.html?q=oauth). For information about how to create a refresh token, see [Generate an OAuth Token Using a Refresh Token](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/generate-an-oauth-token-using-a-refresh-token.html?q=oauth). This is the cURL request for generating an access or refresh token. ```cURL // Generate an access token curl -H "Content-type: application/x-www-form-urlencoded" -X POST https://api.einstein.ai/v2/oauth2/token -d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=" // Generate a refresh token curl -H "Content-type: application/x-www-form-urlencoded" -X POST https://api.einstein.ai/v2/oauth2/token -d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=&scope=offline" ``` #### Request body Media type: `application/x-www-form-urlencoded`. Schema: `object`. * `assertion` (_string_) encrypted payload to identify yourself Example: `SOME_ASSERTION_STRING`. * `scope` (_string_) set to `offline` to generate a refresh token Example: `offline`. * `grant_type` (_string_, required) specify the authentication method desired Allowed values: `urn:ietf:params:oauth:grant-type:jwt-bearer`, `refresh_token`. Example: `urn:ietf:params:oauth:grant-type:jwt-bearer`. * `refresh_token` (_string_) The refresh token you created previously. Example: `SomeRefreshToken`. * `valid_for` (_integer (int32)_) Number of seconds until the access token expires. Default is 60 seconds. Maximum value is 30 days Default: `60`. Example: `120`. #### Responses * **200** Access token result Media type: `application/json`. Schema: `GenerateAccessTokenResponse`. * **400** Bad request ### POST insightJSON `POST /v2/language/analyze` Operation ID: `insightJSON`. Tags: Predictions (Language). #### Request body (required) Conversation turn & historical dialogues Media type: `application/json`. Schema: `ELURequest`. #### Responses * **200** Generate NLU insights Media type: `application/json`. Schema: `ELUResponse`. #### Code samples _Example: cURL · curl_ ``` curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: application/json" -d '{"text":"","locale":"en_US","timeZone":"America/Los_Angeles","context":{"models":[{"id":"","type":"intents","numResults":1}],"dialogue":[{"agent":1,"text":""}]},"agent":1}' "https://api.einstein.ai/v2/language/analyze" ``` ### POST Detect Entities in Text `POST /v2/language/entities` Operation ID: `entitiesMultipart`. Tags: Predictions (Language). The Einstein NER (beta) named-entity recognition (NER) model returns entities for a specified string. **Note:** As a beta feature, Einstein NER is a preview and isn’t part of the “Services” under your Main Services Agreement with Salesforce. Use this feature at your sole discretion, and make your purchase decisions only on the basis of generally available products and features. Salesforce doesn’t guarantee general availability of this feature within any particular time frame or at all, and we can discontinue it at any time. This feature is for evaluation purposes only, not for production use. It’s offered as is and isn’t supported, and Salesforce has no liability for any harm or damage arising out of or in connection with it. All restrictions, Salesforce reservation of rights, obligations concerning the Services, and terms for related Non-Salesforce Applications and Content apply equally to your use of this feature. #### Request Parameters | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `document` | string | Text to classify. Maximum length is 3,000 characters. | 2.0 | | `language` | string | Language of the document parameter. Optional. Valid values: Brazilian Portuguese `pt_BR`; Chinese (Simplified) `zh_CN`; Chinese (Traditional) `zh_TW`; Danish (beta) `da`; Dutch `nl_NL`; English UK `en_GB`; English US `en_US`; French `fr`; German `de`; Italian `it`; Japanese `ja`; Korean (beta) `ko`; Portuguese `pt_PT`; Russian `ru`; Spanish `es`; Swedish (beta) `sv`. | 2.0 | | `modelId` | string | ID of the model. Optional. Defaults to `ENTITIES`. | 2.0 | | `timeZone` | string | Time zone of the returned DATETIME entity. Optional. Defaults to `America/Los_Angeles`. See this list of [tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for valid time zones, and use the TZ database name value from the table. For example, `Africa/Abidjan`. | 2.0 | | `sampleId` | string | String that you can pass in to tag the prediction. Optional. Can be any value, and is returned in the response. | 2.0 | **Note:** As a beta feature, Danish, Korean, and Swedish language support is a preview and isn’t part of the “Services” under your master subscription agreement with Salesforce. Use this feature at your sole discretion, and make your purchase decisions only on the basis of generally available products and features. Salesforce doesn’t guarantee general availability of this feature within any particular time frame or at all, and we can discontinue it at any time. This feature is for evaluation purposes only, not for production use. It’s offered as is and isn’t supported, and Salesforce has no liability for any harm or damage arising out of or in connection with it. All restrictions, Salesforce reservation of rights, obligations concerning the Services, and terms for related Non-Salesforce Applications and Content apply equally to your use of this feature. This is an example of the cURL request with parameters. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "document=Marc Benioff, the CEO of Salesforce, gave the keynote speech at the conference in Paris last week." https://api.einstein.ai/v2/language/entities ``` Keep the following points in mind: * **Response Sort Order**—The entities returned in the response are sorted by the `span` `start` value. This is the start position in the `document` parameter string of the words the the API recognizes as an entity. #### Pass Request Parameters as JSON When you call the NER endpoint, you can specify a Content-Type of `application/json` and pass the request parameters as JSON. ```json { "modelId": "ENTITIES", "language": "en_US", "document": "Marc Benioff, the CEO of Salesforce, gave the keynote speech at the conference in Paris last week." } ``` The cURL call looks like this. _Windows_ ```cURL curl -X POST -H "Authorization: Bearer " -H "Content-Type: application/json" -d "{\"modelId\":\"ENTITIES\", \"language\": \"en_US\", \"document\":\"Marc Benioff, the CEO of Salesforce, gave the keynote speech at the conference in Paris last week.\"}" https://api.einstein.ai/v2/language/entities ``` _Linux_ ```cURL curl -X POST -H "Authorization: Bearer " -H "Content-Type: application/json" -d '{"modelId":"ENTITIES", "language": "en_US", "document":"Marc Benioff, the CEO of Salesforce, gave the keynote speech at the conference in Paris last week."}' https://api.einstein.ai/v2/language/entities ``` #### Request body Media type: `multipart/form-data`. Schema: `EntitiesPredictRequest`. #### Responses * **200** Prediction result Media type: `application/json`. Schema: `EntitiesPredictResponse`. * **429** Exceed usage limitation Media type: `application/json`. Schema: `PredictionErrorResponse`. ### POST Prediction for Intent `POST /v2/language/intent` Operation ID: `intentMultipart`. Tags: Predictions (Language). Returns an intent prediction for the given string. #### Request Parameters | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `document` | string | Text for which you want to return an intent prediction. | 2.0 | | `modelId` | string | ID of the model that makes the prediction. The model must have been created from a dataset with a type of `text-intent`. | 2.0 | | `numResults` | int | Number of probabilities to return. Optional. If passed, must be a number greater than zero. The response is sorted by `probability` in descending order. For example, if you pass in 3, only the top three `label` and `probability` values are returned. | 2.0 | | `sampleId` | string | String that you can pass in to tag the prediction. Optional. Can be any value, and is returned in the response. | 2.0 | This is an example of the cURL request with parameters. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "modelId=WEQ6PHPBGFYVX5C7QDP6XU3NXY" -F "document=what is the weather in los angeles" https://api.einstein.ai/v2/language/intent ``` Keep the following points in mind when sending text in for prediction: * If you omit the `numResults` parameter, the response returns probabilities for all the labels in the model. #### Passing Parameters as JSON As an alternative to passing the `modelId` and `document` parameters as multipart/form-data fields, you can pass those parameters as JSON. You can pass the JSON directly as a string or reference a file that contains the JSON. This cURL call sends a JSON string into a case routing model for classification. The `Content-Type` has changed to `application/json`. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: application/json" -d "{\"modelId\":\"2ELVAO5BNVGZLBHMYTV7CGD5AY\",\"document\":\"my password stopped working\"}" https://api.einstein.ai/v2/language/intent ``` Here's what the response might look like. ```json { "probabilities": [ { "label": "Password Help", "probability": 0.99040705 }, { "label": "Order Change", "probability": 0.003532466 }, { "label": "Shipping Info", "probability": 0.003473858 }, { "label": "Billing", "probability": 0.0024010758 }, { "label": "Sales Opportunity", "probability": 0.00018560764 } ], "object": "predictresponse" } ``` You can store the JSON in a file and reference that file. For example, the file intent.json might look like the following. ```json { "modelId": "2ELVAO5BNVGZLBHMYTV7CGD5AY", "document": "my password stopped working" } ``` This cURL call references that file. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: application/json" -d @c:\data\intent.json https://api.einstein.ai/v2/language/intent ``` ##### Rate Limit Headers Any time you make an API call to the `/intent` resource, your rate limit information is returned in the header. The rate limit headers specify your prediction usage for the current calendar month only. ```text X-RateLimit-Limit 2000 X-RateLimit-Remaining 1997 X-RateLimit-Reset 2017-04-01 19:31:42.0 ``` | Header | Description | Example | | --- | --- | --- | | `X-RateLimit-Limit` | Maximum number of prediction calls available for the current plan month. | 2000 | | `X-RateLimit-Remaining` | Total number of prediction calls you have left for the current plan month. | 1997 | | `X-RateLimit-Reset` | Date on which your predictions are next provisioned. Always the first of the month. | 2017-04-01 22:07:40.0 | #### Request body Media type: `multipart/form-data`. Schema: `IntentPredictRequest`. Media type: `application/json`. Schema: `IntentPredictRequest`. #### Responses * **200** Prediction result Media type: `application/json`. Schema: `IntentPredictResponse`. * **400** Bad request * **404** Not found * **429** Exceed usage limitation Media type: `application/json`. Schema: `PredictionErrorResponse`. ### POST Prediction for Sentiment `POST /v2/language/sentiment` Operation ID: `sentimentMultipart`. Tags: Predictions (Language). Returns a sentiment prediction for the given string. #### Request Parameters | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `document` | string | Text for which you want to return a sentiment prediction. | 2.0 | | `modelId` | string | ID of the model that makes the prediction. The model must have been created from a dataset with a type of `text-sentiment`. | 2.0 | | `numResults` | int | Number of probabilities to return. Optional. If passed, must be a number greater than zero. The response is sorted by `probability` in descending order. For example, if you pass in 3, only the top three `label` and `probability` values are returned. | 2.0 | | `sampleId` | string | String that you can pass in to tag the prediction. Optional. Can be any value, and is returned in the response. | 2.0 | This is an example of the cURL request with parameters. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "modelId=WJH4YCA7YX4PCWVNCYNWYHBMY4" -F "document=I can't tell you how much fun it was" https://api.einstein.ai/v2/language/sentiment ``` Keep the following points in mind when sending text in for prediction: * If you omit the `numResults` parameter, the response returns probabilities for all the labels in the model. #### Passing Parameters as JSON As an alternative to passing the `modelId` and `document` parameters as multipart/form-data fields, you can pass those parameters as JSON. You can pass the JSON directly as a string or reference a file that contains the JSON. This cURL call sends a JSON string into the prebuilt sentiment model for classification. The `Content-Type` has changed to `application/json`. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: application/json" -d "{\"modelId\":\"CommunitySentiment\",\"document\":\"The presentation was great and I learned a lot\"}" https://api.einstein.ai/v2/language/sentiment ``` Here's what the response might look like. ```json { "probabilities": [ { "label": "positive", "probability": 0.8673582 }, { "label": "negative", "probability": 0.1316828 }, { "label": "neutral", "probability": 0.0009590242 } ], "object": "predictresponse" } ``` You can store the JSON in a file and reference that file. For example, the file sentiment.json might look like the following. ```json { "modelId": "CommunitySentiment", "document": "The presentation was great and I learned a lot" } ``` This cURL call references that file. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: application/json" -d @c:\data\sentiment.json https://api.einstein.ai/v2/language/sentiment ``` ##### Rate Limit Headers Any time you make an API call to the `/sentiment` resource, your rate limit information is returned in the header. The rate limit headers specify your prediction usage for the current calendar month only. ```text X-RateLimit-Limit 2000 X-RateLimit-Remaining 1997 X-RateLimit-Reset 2017-04-01 19:31:42.0 ``` | Header | Description | Example | | --- | --- | --- | | `X-RateLimit-Limit` | Maximum number of prediction calls available for the current plan month. | 2000 | | `X-RateLimit-Remaining` | Total number of prediction calls you have left for the current plan month. | 1997 | | `X-RateLimit-Reset` | Date on which your predictions are next provisioned. Always the first of the month. | 2017-04-01 22:07:40.0 | #### Request body Media type: `multipart/form-data`. Schema: `SentimentPredictRequest`. Media type: `application/json`. Schema: `SentimentPredictRequest`. #### Responses * **200** Prediction result Media type: `application/json`. Schema: `SentimentPredictResponse`. * **400** Bad request * **404** Not found * **429** Exceed usage limitation Media type: `application/json`. Schema: `PredictionErrorResponse`. ### GET Get a Dataset `GET /v2/language/datasets/{datasetId}` Operation ID: `getDataset`. Tags: Datasets (Language). Returns a single dataset. #### Parameters * `datasetId` (path, integer (int64), required) Dataset Id _Example_ ``` SomeDatasetId ``` #### Responses * **200** Success Media type: `application/json`. Schema: `Dataset`. * **404** Not found #### Code samples _Example: cURL · curl_ ``` curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/language/datasets/" ``` ### DELETE Delete a Dataset `DELETE /v2/language/datasets/{datasetId}` Operation ID: `deleteDataset`. Tags: Datasets (Language). Deletes the specified text dataset and its labels and examples. This is an example of the cURL request. ```cURL curl -X DELETE -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/datasets/10008 ``` Keep the following points in mind when deleting a dataset. * If a dataset is being trained and has an associated model with a status of `QUEUED` or `RUNNING`, you must wait until the training is complete before you can delete the dataset. * If you want to delete a dataset and the models associated with it, delete the models first. Deleting a dataset doesn't delete the models created from that dataset. After you delete a dataset, you can still make predictions against the associated models, but you lose the ability to list the models for the deleted dataset. **Important:** As of August 14, 2023 Get Deletion Status has been deprecated and will no longer be supported by Salesforce. Deletion is now immediate. To determine if the a dataset has been deleted use the Get Dataset command. A deleted dataset will return a 404 (Not Found) http status. * After you delete a dataset, use the `id` to get the status of the deletion. See Get Deletion Status. #### Parameters * `datasetId` (path, integer (int64), required) Dataset Id _Example_ ``` SomeDatasetId ``` #### Responses * **200** Success, deletion job created Media type: `application/json`. Schema: `DeletionResponse`. * **404** Not found ### GET Get All Examples `GET /v2/language/datasets/{datasetId}/examples` Operation ID: `getExamples`. Tags: Examples (Language). Returns all the examples for the specified dataset. This is an example of the cURL request. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/datasets/57/examples ``` #### Page Through Examples By default, this call returns 100 examples. If you want to page through the examples in a dataset, use the `offset` and `count` query parameters. | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `count` | int | Number of examples to return. Maximum valid value is 100. If you specify a number greater than 100, the call returns 100 examples. Optional. | 1.0 | | `offset` | int | Index of the example from which you want to start paging. Optional. | 1.0 | Here's an example of these query parameters. If you omit the `count` parameter or the `count` parameter is greater than 100, the API returns 100 examples. If you omit the `offset` parameter, paging starts at 0. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/language/datasets/57/examples?offset=100&count=50" ``` ##### How Paging Works To page through all the examples in a dataset: 1. Make the Get a Dataset call to return the `totalExamples` value for the dataset. 2. Make the Get All Examples call and pass in the `offset` and `count` values until you reach the end of the examples. For example, let's say you have a dataset and you want to display information about the examples in a UI and show 15 at a time. The first call would have `offset=0` and `count=15`, the second call would have `offset=15` and `count=15`, and so on. #### Return Specific Example Types Use the `source` query parameter to return examples that were created in the dataset. * `all`—Return upload examples. * `upload`—Return examples that were created from uploading a file. #### Limits Each Einstein Vision and Language account is limited to 30 calls per calendar month to Einstein Language endpoints that return examples. This limit applies across all APIs that return examples. If you exceed this limit, you receive an error message. See [Release Notes](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/release-notes-einstein-platform-services.html). #### Parameters * `datasetId` (path, integer (int64), required) Dataset Id _Example_ ``` SomeDatasetId ``` * `offset` (query, integer (int32)) Index of the example from which you want to start paging. Default: `0`. * `count` (query, integer (int32)) Number of examples to return. Default: `100`. * `source` (query, string) return examples that were created in the dataset Allowed values: `all`, `upload`. #### Responses * **200** Success Media type: `application/json`. Schema: `ExampleList`. * **400** Bad request * **404** Dataset not found ### GET Get All Examples for Label `GET /v2/language/examples/{datasetId}/label/{labelId}` Operation ID: `getExamplesByLabel`. Tags: Predictions (Language). Returns all the examples for the specified label. Returns uploaded examples. **NOTE:** This is a new endpoint. Use this updated endpoint after December 3, 2020 to get all examples for a label. This is an example of the cURL request. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/examples/1000108/label/16599 ``` #### Page Through Examples By default, this call returns 100 examples. If you want to page through all the examples returned by this call, use the `offset` and `count` query parameters. The `numExamples` value indicates how many examples have the specified label, so you can use this value to control the paging. | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `count` | int | Number of examples to return. Maximum valid value is 100. If you specify a number greater than 100, the call returns 100 examples. Optional. | 1.0 | | `offset` | int | Index of the example from which you want to start paging. Optional. | 1.0 | The following call shows how to page through examples using these query parameters. If you omit the `count` parameter or the `count` parameter is greater than 100, the API returns 100 examples. If you omit the `offset` parameter, paging starts at 0. When using multiple query parameters in a cURL call, be sure to enclose the endpoint in quotes. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/language/examples/7890/label/1234?offset=50&count=50" ``` **Note:** This call returns uploaded examples that have the specified label. This call doesn't support the `source` parameter. #### Limits Each Einstein Vision and Language account is limited to 30 calls per calendar month to Einstein Language endpoints that return examples. This limit applies across all APIs that return examples. If you exceed this limit, you receive an error message. See [Release Notes](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/release-notes-einstein-platform-services.html). #### Parameters * `datasetId` (path, integer (int64), required) Dataset Id _Example_ ``` SomeDatasetId ``` * `labelId` (path, integer (int32), required) Label Id _Example_ ``` SomeLabelId ``` * `offset` (query, integer (int32)) Index of the example from which you want to start paging. Default: `0`. * `count` (query, integer (int32)) Number of examples to return. Default: `100`. #### Responses * **200** Success Media type: `application/json`. Schema: `ExampleList`. * **400** Bad request * **404** Dataset not found ### GET Get All Models `GET /v2/language/datasets/{datasetId}/models` Operation ID: `getTrainedModels`. Tags: Models (Language). Returns all models for the specified dataset. This is an example of the cURL request. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/datasets/1001511/models ``` #### Page Through Models By default, this call returns 100 models. If you want to page through your models, use the `offset` and `count` query parameters. | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `count` | int | Number of models to return. Maximum valid value is 100. If you specify a number greater than 100, the call returns 100 models. Optional. | 1.0 | | `offset` | int | Index of the dataset from which you want to start paging. Optional. | 1.0 | Here's an example of these query parameters. If you omit the `offset` parameter, paging starts at 0. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/language/datasets//models?offset=5&count=5" ``` #### Parameters * `offset` (query, integer (int32)) Index of the model from which you want to start paging. Default: `0`. * `count` (query, integer (int32)) Number of models to return. Default: `100`. * `datasetId` (path, string, required) Dataset Id _Example_ ``` SomeDatasetId ``` #### Responses * **200** Success Media type: `application/json`. Schema: `ModelList`. * **404** Not found ### GET Get All Datasets `GET /v2/language/datasets` Operation ID: `listDatasets`. Tags: Datasets (Language). Returns a list of datasets and their labels that were created by the current user. The response is sorted by dataset ID. This is an example of the cURL request. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/datasets ``` #### Page Through Datasets By default, this call returns 25 datasets. If you want to page through your datasets, use the `offset` and `count` query parameters. | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `count` | integer | Number of datsets to return. Maximum valid value is 25. If you specify a number greater than 25, the call returns 25 datasets. Optional. | 2.0 | | `offset` | integer | Index of the dataset from which you want to start paging. Optional. | 2.0 | Here's an example of these query parameters. If you omit the `count` parameter, the API returns 25 datasets. If you omit the `offset` parameter, paging starts at 0. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/language/models/TUWBAIRLFB3KKEDZMEGIUR6COM/lc?offset=100&count=20" ``` For example, let's say you want to page through all of your datasets and show 20 at a time. The first call would have `offset=0` and `count=20`, the second call would have `offset=20` and `count=20`, and so on. #### Get Global Datasets Global datasets are public datasets that Salesforce provides. You can use these datasets to include additional data during training when you create a model. To get a list of the global datasets, use the `global` query parameter. | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `global` | boolean | If `true`, returns all global datasets. | 2.0 | Here's an example of the `global` query parameter. The response JSON is the same as for your own custom datasets. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/language/datasets?global=true" ``` #### Parameters * `count` (query, integer (int32)) Number of datsets to return. Maximum valid value is 25. If you specify a number greater than 25, the call returns 25 datasets. Default: `25`. * `offset` (query, integer (int32)) Index of the dataset from which you want to start paging Default: `0`. * `global` (query, boolean) If true, returns all global datasets. Global datasets are public datasets that Salesforce provides. Default: `false`. #### Responses * **200** Success Media type: `application/json`. Schema: `DatasetList`. ### PUT Create Examples From a File `PUT /v2/language/datasets/{datasetId}/upload` Operation ID: `updateDatasetAsync`. Tags: Examples (Language). Adds examples from a .csv, .tsv, or .json file to a dataset. #### Request Parameters | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `data` | string | Path to the .csv, .tsv, or .json file on a local drive. The maximum file size you can upload from a local drive is 25 MB. | 2.0 | Here's an example of the cURL request with parameters. ```cURL // Add examples to a dataset from a local file curl -X PUT -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "data=@C:\Data\weather_update.csv" https://api.einstein.ai/v2/language/datasets/1001412/upload ``` This call adds examples to the specified dataset from a .csv, .tsv, or .json file. This is an asynchronous call, so the results that are initially returned contain information for the original dataset and `available` is `false`. Use the dataset ID and use Get a Dataset to query when the upload is complete. When `available` is `true` and `statusMsg` is `SUCCEEDED`, the data upload is complete. Each dataset type supports different file formats. This table lists the file formats supported by each dataset type. | | text-intent | text-sentiment | | --- | --- | --- | | **.csv file** | Y | Y | | **.tsv file** | Y | Y | | **.json file** | Y | N | Keep the following tipd in mind when you create examples: **FILE SIZE** * The maximum file size you can upload from a local drive is 25 MB. **DATASETS** * The maximum total dataset size is 2 GB. **LABELS** * If the file contains a label that's already in the dataset, the API adds the sentiment or intent strings (examples) with the specified label in the dataset. * If the file contains a label that isn't in the dataset, the API adds a new label (label name limit is 180 characters). **EXAMPLES** * A dataset can have a maximum of 3 million words across all examples. If you try to train a dataset that has more than 3 million words, you receive an error. * For best results, we recommend that each example is around 100 words. **OTHER** * The Einstein Language APIs support only UTF-8 text characters. If your examples or labels contain any non-UTF-8 text, you receive an error that the file format is invalid when you try to create the examples. * If you try to create examples in a dataset while a previous call to create examples is still processing (the dataset's `available` value is `false`), the call fails and you receive an error. You must wait until the dataset's `available` value is `true` before starting another upload. See Create a Dataset From a File Asynchronously or Create a Dataset From a File Synchronously for information about the file structure. #### Parameters * `datasetId` (path, integer (int64), required) Dataset Id _Example_ ``` SomeDatasetId ``` #### Request body Media type: `multipart/form-data`. Schema: `object`. * `data` (_string_) Path to the .csv, .tsv, or .json file on a local drive. * `path` (_string_) URL of the .csv, .tsv, or .json file. * `type` (_string_) Path of the .csv, .tsv, or .json file. #### Responses * **200** Upload success Media type: `application/json`. Schema: `Dataset`. ### POST Create a Dataset From a File Asynchronously `POST /v2/language/datasets/upload` Operation ID: `uploadDatasetAsync`. Tags: Datasets (Language). Creates a dataset, labels, and examples from the specified .csv, .tsv, or .json file. The call returns immediately and continues to upload data in the background. #### Request Parameters | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `data` | string | Path to the .csv, .tsv, or .json file on the local drive (FilePart). The maximum file size you can upload from a local drive is 25 MB. | 2.0 | | `language` | string | (Optional) Dataset language. Default is `en_US`. See [Supported Languages](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/supported-languages.html?q=supported%20languages). | 2.0 | | `name` | string | (Optional) Name of the dataset. If this parameter is omitted, the dataset name is derived from the file name. | 2.0 | | `type` | string | Type of dataset data. Valid values are `text-intent` and `text-sentiment`. | 2.0 | Here's an example of the cURL request with parameters. ```cURL // Create a dataset from a local file curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "data=@C:\Data\weather.csv" -F "type=text-intent" https://api.einstein.ai/v2/language/datasets/upload ``` The API call is asynchronous, so you receive a dataset ID back immediately but the `available` value is `false` and the `statusMsg` value is `UPLOADING`. Use the dataset ID and make a call to Get a Dataset to query when the upload is complete. When `available` is `true` and `statusMsg` is `SUCCEEDED`, the data upload is complete, and you can train the dataset to create a model. Keep the following points in mind when creating datasets. **FILE SIZE** * The maximum file size you can upload from a local drive or web location is 25 MB. * If your file is more than 20 MB, for better performance, we recommend that you upload it to a cloud location that doesn't require authentication and pass the URL in the `path` parameter. * If you have a large amount of data, consider breaking up your data into multiple files. You can load the first file using this call and then load subsequent files using PUT. See Create Examples From a File. **DATASETS** * The maximum total dataset size is 2 GB. **LABELS** * The maximum label name length is 180 characters. If a file contains a class label name greater than 180 characters, the label is created in the dataset, but the API truncates the label name to 180 characters. * You must have at least two labels in the dataset. If you don't have at least two labels, you can create the dataset but training the dataset fails. * A dataset can have up to 500 labels. However, we recommend that a dataset has a maximum of 100 labels for better model accuracy. **EXAMPLES** * If a file contains duplicate intent or sentiment strings, only the first one is loaded. * If the dataset type is `text-intent`, each label must have at least 20 examples. We recommend that each label has at least 100 examples for better model accuracy. If you have only 20 examples, be sure to use the `multilingual-intent` or `multilingual-intent-ood` algorithm when you train the dataset. * If the dataset type is `text-sentiment`, each label must have at least 100 examples. * We recommend a maximum of 10,000 examples across all labels. * A dataset can have a maximum of 3 million words across all examples. If you try to train a dataset that has more than 3 million words, you receive an error. * For best results, we recommend that each example is less than 150 words. * If the file contains duplicate text strings (examples), only the first string is uploaded. If there's more than one text string with the same text, only the first string is uploaded and the others are skipped. This is true whether the duplicate text strings have the same label or different labels. **OTHER** * The Einstein Language APIs support only UTF-8 text characters. If your examples or labels contain any non-UTF-8 text, you receive an error that the file format is invalid when you try to create the dataset. Each dataset type supports different file formats. This table lists the file formats supported by each dataset type. | | text-intent | text-sentiment | | --- | --- | --- | | **.csv file** | Y | Y | | **.tsv file** | Y | Y | | **.json file** | Y | N | #### CSV File Format Considerations * You can use a .csv file to create both intent and sentiment datasets. * When you create a dataset from a .csv file, the dataset name is inherited from the file name. * Each .csv file contains sentiment or intent data in this format: `"sentiment or intent string", label-name` followed by a CRLF. In the following example, the intent string is in double quotes followed by the label `current-weather`. ```text "what's the weather look like",current-weather "is it raining",current-weather "what's the temperature",current-weather ``` #### TSV File Format Considerations * You can use a .tsv file to create both intent and sentiment datasets. * When you create a dataset from a .tsv file, the dataset name is inherited from the file name. * Each .tsv file contains sentiment or intent data in this format: `"sentiment or intent string"label-name` followed by a CRLF. In the following example, the intent string is in double quotes followed by the label `current-weather`. ```text "what's the weather look like" current-weather "is it raining" current-weather "what's the temperature" current-weather ``` #### JSON File Format Considerations * You can use a .json file to create only an intent dataset. * The top-level object must be called `"intents"` as shown in the example file. * When you create a dataset from a .json file, the dataset name is inherited from the file name. * The JSON structure is a top-level object that contains multiple arrays. Each array contains the intent strings in double quotes. This image shows the structure of the JSON file. The labels are `current-weather`, `five-day-forecast`, and `hourly-forecast`. The content in double quotes becomes a single example that has the specified label. #### Request body Media type: `multipart/form-data`. Schema: `object`. * `data` (_string_) Path to the .csv, .tsv, or .json file on the local drive (FilePart). * `name` (_string_) Name of the dataset. Optional. If this parameter is omitted, the dataset name is derived from the file name. Default: `dataset`. Example: `weather`. * `path` (_string_) URL of the .csv, .tsv, or .json file. * `language` (_string_) dataset language Default: `en_US`. * `type` (_string_, required) Type of dataset data. Allowed values: `text-intent`, `text-sentiment`. * `entityName` (_string_) Entity name, optional. Minimum length: 0. Maximum length: 180. * `entityFilter` (_string_) Entity filter, optional. Minimum length: 0. Maximum length: 4096. #### Responses * **200** Upload initiated Media type: `application/json`. Schema: `Dataset`. * **400** Bad request ### POST Create a Dataset From a File Synchronously `POST /v2/language/datasets/upload/sync` Operation ID: `uploadDatasetSync`. Tags: Datasets (Language). Creates a dataset, labels, and examples from the specified .csv, .tsv, or .json file. The call returns after the dataset is created and all of the data is uploaded. Use this API call for files that are smaller than 10 MB. ## Request Parameters | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `data` | string | Path to the .csv, .tsv, or .json file on the local drive (FilePart). The maximum file size you can upload from a local drive is 25 MB. | 2.0 | | `language` | string | (Optional) Dataset language. Default is `en_US`. See [Supported Languages](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/supported-languages.html?q=supported%20languages). | 2.0 | | `name` | string | (Optional) Name of the dataset. If this parameter is omitted, the dataset name is derived from the file name. | 2.0 | | `type` | string | Type of dataset data. Valid values are: `text-intent` and `text-sentiment` | 2.0 | Here's an example of the cURL request with parameters. ```cURL // Create a dataset from a local file curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "data=@C:\Data\weather.csv" -F "type=text-intent" https://api.einstein.ai/v2/language/datasets/upload/sync ``` The API call is synchronous, so results are returned after the data has been uploaded to the dataset. If this call succeeds, it returns the `labels` array, `available` is `true`, and `statusMsg` is `SUCCEEDED`. Keep the following points in mind when creating datasets. **FILE SIZE** * The maximum file size you can upload from a local drive or web location is 25 MB. * If your file is larger than 10 MB, we recommend that you use the asynchronous call to create a dataset. If you use this call with a large dataset .zip file, the call could time out. See Create a Dataset From a File Asynchronously. * If you have a large amount of data, consider breaking up your data into multiple files. You can load the first file using this call and then load subsequent files using PUT. See Create Examples From a File. **DATSETS** * The maximum total dataset size is 2 GB. **LABELS** * The maximum label name length is 180 characters. If a file contains a class label name greater than 180 characters, the label is created in the dataset, but the API truncates the label name to 180 characters. * You must have at least two labels in the dataset. If you don't have at least two labels, you can create the dataset but training the dataset fails. * A dataset can have up to 500 labels. However, we recommend that a dataset has a maximum of 100 labels for better model accuracy. **EXAMPLES** * If a file contains duplicate intent or sentiment strings, only the first one is loaded. * If the dataset type is `text-intent`, each label must have at least 20 examples. We recommend that each label has at least 100 examples for better model accuracy. If you have only 20 examples, be sure to use the `multilingual-intent` or `multilingual-intent-ood` algorithm when you train the dataset. * If the dataset type is `text-sentiment`, each label must have at least 100 examples. * We recommend a maximum of 10,000 examples across all labels. * A dataset can have a maximum of 3 million words across all examples. If you try to train a dataset that has more than 3 million words, you receive an error. * For best results, we recommend that each example is less than 150 words. \-If the file contains duplicate text strings, only the first string is uploaded. If there's more than one text string with the same text, only the first string is uploaded and the others are skipped. This is true whether the duplicate text strings have the same label or different labels. **OTHER** * The Einstein Language APIs support only UTF-8 text characters. If your examples or labels contain any non-UTF-8 text, you receive an error that the file format is invalid when you try to create the dataset. Each dataset type supports different file formats. This table lists the file formats supported by each dataset type. | | text-intent | text-sentiment | | --- | --- | --- | | **.csv file** | Y | Y | | **.tsv file** | Y | Y | | **.json file** | Y | N | #### CSV File Format Considerations * You can use a .csv file to create both intent and sentiment datasets. * When you create a dataset from a .csv file, the dataset name is inherited from the file name. * Each .csv file contains sentiment or intent data in this format: `"sentiment or intent string", label-name` followed by a CRLF. In the following example, the intent string is in double quotes followed by the label `current-weather`. ```text "what's the weather look like",current-weather "is it raining",current-weather "what's the temperature",current-weather ``` #### TSV File Format Considerations * You can use a .tsv file to create both intent and sentiment datasets. * When you create a dataset from a .tsv file, the dataset name is inherited from the file name. * Each .tsv file contains sentiment or intent data in this format: `"sentiment or intent string"label-name` followed by a CRLF. In the following example, the intent string is in double quotes followed by the label `current-weather`. ```text "what's the weather look like" current-weather "is it raining" current-weather "what's the temperature" current-weather ``` #### JSON File Format Considerations * You can use a .json file to create only an intent dataset. * The top-level object must be called `"intents"` as shown in the example file. * When you create a dataset from a .json file, the dataset name is inherited from the file name. * The JSON structure is a top-level object that contains multiple arrays. Each array contains the intent strings in double quotes. This image shows the structure of the JSON file. The labels are `current-weather`, `five-day-forecast`, and `hourly-forecast`. The content in double quotes becomes a single example that has the specified label. #### Request body Media type: `multipart/form-data`. Schema: `object`. * `data` (_string_) Path to the .csv, .tsv, or .json file on the local drive (FilePart). * `name` (_string_) Name of the dataset. Optional. If this parameter is omitted, the dataset name is derived from the file name. Example: `weather`. * `path` (_string_) URL of the .csv, .tsv, or .json file. * `language` (_string_) dataset language Default: `en_US`. * `type` (_string_) Type of dataset data. Allowed values: `text-intent`, `text-sentiment`. #### Responses * **200** Upload success Media type: `application/json`. Schema: `Dataset`. * **400** Bad request ### GET Get Model Metrics `GET /v2/language/models/{modelId}` Operation ID: `getTrainedModelMetrics`. Tags: Models (Language). Returns the metrics for a model, such as the f1 score, accuracy, and confusion matrix. The combination of these metrics gives you a picture of model accuracy and how well the model will perform. This call returns the metrics for the last epoch in the training used to create the model. This is an example of the cURL request. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/models/TUWBAIRLFB3KKEDZMEGIUR6COM ``` Keep the following points in mind when analyzing the precision-recall metrics. * The precision-recall curve arrays are calculated using the examples in the test set (as specified by the `trainSplitRatio`). The margin of error when these values are computed depends on the number of examples in the test set. * The granularity of the threshold variations depends on the number of examples in the test set and the number of labels in the dataset from which the model was created. * The maximum number of values returned in any one of these arrays is 2,000. #### Parameters * `modelId` (path, string, required) Model Id _Example_ ``` SomeModelId ``` #### Responses * **200** Model metrics Media type: `application/json`. Schema: `Metrics`. * **404** Resource not found * **502** Unable to complete request due to service error ### DELETE Delete a Model `DELETE /v2/language/models/{modelId}` Operation ID: `deleteModel`. Tags: Models (Language). Deletes the specified model. After you delete a model, you can use the `id` to get the status of the deletion. See Get Deletion Status. This is an example of the cURL request. ```cURL curl -X DELETE -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/models/KD33Y2UZHI2TDB7VXQSUHZBNGA ``` #### Parameters * `modelId` (path, string, required) Model Id _Example_ ``` SomeModelId ``` #### Responses * **201** Deletion submitted Media type: `application/json`. Schema: `DeletionResponse`. * **404** Resource not found * **502** Unable to complete request due to service error ### GET Get Training Status `GET /v2/language/train/{modelId}` Operation ID: `getTrainStatusAndProgress`. Tags: Training (Language). Returns the status of a training job. Use the progress field to determine how far the training has progressed. When training completes successfully, the `status` is `SUCCEEDED` and the `progress` is `1`. This is an example of the cURL request. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/train/TUWBAIRLFB3KKEDZMEGIUR6COM ``` #### Parameters * `modelId` (path, string, required) Model Id _Example_ ``` SomeModelId ``` #### Responses * **200** Training status Media type: `application/json`. Schema: `TrainResponse`. * **400** Bad request * **404** Resource not found * **502** Unable to complete request due to service error ### GET Get Model Learning Curve `GET /v2/language/models/{modelId}/lc` Operation ID: `getTrainedModelLearningCurve`. Tags: Models (Language). Returns the metrics for each epoch in a model. These metrics show you the f1 score, accuracy, confusion matrix, test accuracy, and so on for each training iteration performed to create the model. This is an example of the cURL request. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/language/models/TUWBAIRLFB3KKEDZMEGIUR6COM/lc?offset=0&count=3" ``` Use the `labels` array and the `confusionMatrix` array to build the confusion matrix for an epoch. The labels in the array become the matrix rows and columns. Here's what the confusion matrix for the first epoch in the results. | | hourly-forecast | current-weather | five-day-forecast | | --- | --- | --- | --- | | **hourly-forecast** | 0 | 2 | 1 | | **current-weather** | 0 | 8 | 0 | | **five-day-forecast** | 0 | 1 | 0 | #### Page Through Results By default, this call returns learning curve data for 25 epochs. If you want to page through the results, use the `offset` and `count` query parameters. | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `count` | int | Number of epochs for which to return metrics. Maximum valid value is 25. If you specify a number greater than 25, the call returns metrics for 25 epochs. Optional. | 2.0 | | `offset` | int | Index of the epoch from which you want to start paging. Optional. | 2.0 | Here's an example of these query parameters. If you omit the `count` parameter, the API returns 25 epochs. If you omit the `offset` parameter, paging starts at 0. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/language/datasets?offset=100&count=20" ``` For example, let's say you want to page through all of the learning curve results and show 20 at a time. The first call would have `offset=0` and `count=20`, the second call would have `offset=20` and `count=20`, and so on. If you want to find out the total number of epochs that were performed to create a model, make the call the get the training status and use the `lastEpochDone` value. This value will give you an idea of how much paging you need to do. See Get Training Status. #### Parameters * `modelId` (path, string, required) Model Id _Example_ ``` SomeModelId ``` * `offset` (query, integer (int32)) Index of the epoch from which you want to start paging Default: `0`. * `count` (query, integer (int32)) Number of epoch to return. Maximum valid value is 25. Default: `25`. #### Responses * **200** Learning curve Media type: `application/json`. Schema: `LearningCurveList`. * **404** Resource not found * **502** Unable to complete request due to service error ### POST Retrain a Dataset `POST /v2/language/retrain` Operation ID: `retrain`. Tags: Training (Language). Retrains a dataset and updates a model. Use this API call when you want to update a model and keep the model ID instead of creating a new model. #### Request Parameters | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `algorithm` | string | Specifies the algorithm used to train the dataset. Optional. Use this parameter only when training a dataset with a type of `text-intent`. Valid values are `intent` (same as `multilingual-intent`, retained for backward compatibility), `multilingual-intent` (supports multiple languages), and `multilingual-intent-ood` (supports multiple languages and handles out-of-domain text). | 2.0 | | `epochs` | int | Number of training iterations for the neural network. Optional. If not specified, the default is calculated based on the dataset size. The larger the number, the longer the training takes to complete. The training process stops before the specified number of epochs if the model has reached the optimal accuracy. When you get the training staus, the `earlyStopping` field specifies whether the training stopped early, and the `lastEpochDone` value specifies the last training iteration. | 2.0 | | `learningRate` | float | N/A for intent or sentiment models. | 2.0 | | `modelId` | string | ID of the model to be used from the training. | 2.0 | | `trainParams` | object | JSON that contains parameters that specify how the model is created. Optional. Valid values include `{"trainSplitRatio": 0.n}` to specify the ratio of data used to train and test the model, and `{"withGlobalDatasetId": }` to use a global dataset in addition to the specified dataset. | 2.0 | This is an example of the cURL request with parameters. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "modelId=TUWBAIRLFB3KKEDZMEGIUR6COM" https://api.einstein.ai/v2/language/retrain ``` This call retrains the dataset associated with model you specify in the request parameters. Use this call to retrain a dataset and update the model after new examples are added to a dataset. To find out the values specified in the `trainParams` parameter when the model was trained, such as `withGlobalDatasetId`, see Get Training Status. Keep the following points in mind when training a dataset: * A dataset can have only one training in progress at a time. Let's say you retrain a dataset and there's a model with a status of `RUNNING` or `QUEUED`. If you attempt to retrain the same dataset again, you receive an error. * You receive an error when you train a dataset that has more than 3 million words across all examples. Be sure that when you create a dataset or add examples to a dataset, that it contains less than 3 million words. For best results, we recommend that each example is around 100 words. This cURL command sends in the `trainParams` request parameter. This command has double quotes and escaped double quotes around `trainSplitRatio` to run on Windows. You might need to reformat it to run on another OS. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "modelId=TUWBAIRLFB3KKEDZMEGIUR6COM" -F "trainParams={\"trainSplitRatio\":0.7}" https://api.einstein.ai/v2/language/retrain ``` #### Request body Media type: `multipart/form-data`. Schema: `object`. * `modelId` (_string_, required) ID of the model to be updated from the training. Example: `7JXCXTRXTMNLJCEF2DR5CJ46QU`. * `epochs` (_integer (int32)_) Number of training iterations for the neural network. Optional. Minimum: 1. Maximum: 1000. Example: `20`. * `learningRate` (_number (double)_) N/A for intent or sentiment models. Example: `0.0001`. * `trainParams` (_string_) JSON that contains parameters that specify how the model is created * `algorithm` (_string_) Algorithm used for train Default: `multilingual-intent`. Example: `multilingual-intent`. #### Responses * **200** Training status Media type: `application/json`. Schema: `TrainResponse`. * **400** Bad request * **404** Resource not found * **502** Unable to complete request due to service error ### POST Train a Dataset `POST /v2/language/train` Operation ID: `train`. Tags: Training (Language). Trains a dataset and creates a model. #### Request Parameters | Name | Type | Description | Available Version | | --- | --- | --- | --- | | `algorithm` | string | Specifies the algorithm used to train the dataset. Optional. Use this parameter only when training a dataset with a type of `text-intent`. Valid values are `intent` (same as `multilingual-intent`, retained for backward compatibility), `multilingual-intent` (supports multiple languages), and `multilingual-intent-ood` (supports multiple languages and handles out-of-domain text). | 2.0 | | `datasetId` | long | ID of the dataset to train. | 2.0 | | `epochs` | int | Number of training iterations for the neural network. Optional. If not specified, the default is calculated based on the dataset size. The larger the number, the longer the training takes to complete. The training process stops before the specified number of epochs if the model has reached the optimal accuracy. When you get the training staus, the `earlyStopping` field specifies whether the training stopped early, and the `lastEpochDone` value specifies the last training iteration. | 2.0 | | `learningRate` | float | N/A for intent or sentiment models. | 2.0 | | `name` | string | Name of the model. Maximum length is 180 characters. | 2.0 | | `trainParams` | object | JSON that contains parameters that specify how the model is created. Optional. Valid values include `{"trainSplitRatio": 0.n}` to specify the ratio of data used to train and test the model, and `{"withGlobalDatasetId": }` to use a global dataset in addition to the specified dataset. | 2.0 | This is an example of the cURL request with parameters. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Weather Intent Model" -F "datasetId=1001411" https://api.einstein.ai/v2/language/train ``` Keep the following points in mind when training a dataset: * If you’re unsure which values to set for the `epochs` and `learningRate` parameters, we recommend that you omit them and use the defaults. * A dataset can have only one training in progress at a time. Let's say you train a dataset and there's a model with a status of `RUNNING` or `QUEUED`. If you attempt to train the same dataset again, you receive an error. * You receive an error when you train a dataset that has more than 3 million words across all examples. Be sure that when you create a dataset or add examples to a dataset, that it contains less than 3 million words. For best results, we recommend that each example is around 100 words. This cURL command sends in the `trainParams` request parameter. This command has double quotes and escaped double quotes around `trainSplitRatio` to run on Windows. You might need to reformat it to run on another OS. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Weather Intent Model" -F "datasetId=1001411" -F "trainParams={\"trainSplitRatio\":0.7}" https://api.einstein.ai/v2/language/train ``` If you want to train a dataset and update an existing model, see Retrain a Dataset. #### Request body Media type: `multipart/form-data`. Schema: `object`. * `datasetId` (_integer (int64)_, required) ID of the dataset to train. Example: `57`. * `epochs` (_integer (int32)_) Number of training iterations for the neural network. Optional. Minimum: 1. Maximum: 1000. Example: `20`. * `learningRate` (_number (double)_) N/A for intent or sentiment models. * `name` (_string_) Name of the model. Maximum length is 180 characters. Minimum length: 0. Maximum length: 180. * `trainParams` (_string_) JSON that contains parameters that specify how the model is created * `algorithm` (_string_) Algorithm used for train Default: `multilingual-intent`. Example: `multilingual-intent`. #### Responses * **200** Training status Media type: `application/json`. Schema: `TrainResponse`. * **400** Bad request * **404** Resource not found * **502** Unable to complete request due to service error ### POST Detect Text `POST /v2/vision/ocr` Operation ID: `ocrMultipart`. Tags: Vision Prediction. Returns a prediction from an OCR model for the specified image URL or local image file. #### Request body Media type: `multipart/form-data`. Schema: `object`. * `modelId` (_string_, required) ID of the model that makes the prediction. Valid values are OCRModel and tabulatev2. Example: `WJH4YCA7YX4PCWVNCYNWYHBMY4`. * `task` (_string_) Optional. Designates the type of data in the image. Default is text. Valid values: contact, table, and text. Default: `text`. Example: `table`. * `sampleId` (_string_) String that you can pass in to tag the prediction. Optional. Can be any value, and is returned in the response. * `sampleContent` (_string (binary)_) Binary content of image file uploaded as multipart/form-data. Optional. * `sampleLocation` (_string_) URL of the image file. Use this parameter when sending in a file from a web location. Optional. #### Responses * **200** OCR result Media type: `application/json`. Schema: `OCRPredictResponse`. * **400** Bad request * **404** Not found #### Code samples _Example: cURL · curl_ ``` curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "modelId=OCRModel" -F "task=table" -F "sampleId=" -F "sampleContent=@" "https://api.einstein.ai/v2/vision/ocr" ``` ### GET Get a Dataset `GET /v2/vision/datasets/{datasetId}` Operation ID: `getDataset_1`. Tags: Vision Prediction. Returns a single dataset. #### Parameters * `datasetId` (path, integer (int64), required) Dataset Id _Example_ ``` 123456 ``` #### Responses * **200** Success Media type: `application/json`. Schema: `Dataset`. * **404** Not found #### Code samples _Example: cURL · curl_ ``` curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/vision/datasets/" ``` ### DELETE Delete a Dataset `DELETE /v2/vision/datasets/{datasetId}` Operation ID: `deleteDataset_1`. Tags: Vision Prediction. Deletes the specified dataset and associated labels and examples. #### Parameters * `datasetId` (path, integer (int64), required) Dataset Id _Example_ ``` 123456 ``` #### Responses * **200** Success, deletion job created Media type: `application/json`. Schema: `DeletionResponse`. * **404** Not found #### Code samples _Example: cURL · curl_ ``` curl -X DELETE -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/vision/datasets/" ``` ### GET Get All Examples `GET /v2/vision/datasets/{datasetId}/examples` Operation ID: `getExamples_1`. Tags: Vision Prediction. Returns all the examples for the specified dataset. By default, returns examples created by uploading them from a .zip file. #### Parameters * `datasetId` (path, integer (int64), required) Dataset Id _Example_ ``` SomeDatasetId ``` * `offset` (query, integer (int32)) Index of the example from which you want to start paging. Default: `0`. * `count` (query, integer (int32)) Number of examples to return. Default: `100`. * `source` (query, string) return examples that were created in the dataset as feedback Allowed values: `all`, `feedback`, `upload`. #### Responses * **200** Success Media type: `application/json`. Schema: `ExampleList`. * **404** Not found #### Code samples _Example: cURL · curl_ ``` curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/vision/datasets//examples" ``` ### GET Get All Examples for Label `GET /v2/vision/examples/{datasetId}/label/{labelId}` Operation ID: `getExamplesByLabel_1`. Tags: Vision Prediction. Returns all the examples for the specified label. Returns both uploaded examples and feedback examples. #### Parameters * `datasetId` (path, integer (int64), required) Dataset Id _Example_ ``` SomeDatasetId ``` * `labelId` (path, integer (int32), required) Label Id _Example_ ``` SomeLabelId ``` * `offset` (query, integer (int32)) Index of the example from which you want to start paging. Default: `0`. * `count` (query, integer (int32)) Number of examples to return. Default: `100`. #### Responses * **200** Success Media type: `application/json`. Schema: `ExampleList`. * **404** Not found #### Code samples _Example: cURL · curl_ ``` curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/vision/examples//label/" ``` ### GET Get All Datasets `GET /v2/vision/datasets` Operation ID: `listDatasets_1`. Tags: Vision Prediction. Returns All datasets. #### Parameters * `count` (query, integer (int32)) Number of datasets to return. Maximum valid value is 25.If you specify a number greater than 25, the call returns 25 datasets. Default: `25`. * `offset` (query, integer (int32)) Index of the dataset from which you want to start paging. Default: `0`. * `global` (query, boolean) If true, returns all global datasets. Global datasets are public datasets that Salesforce provides. Default: `false`. #### Responses * **200** Success Media type: `application/json`. Schema: `DatasetList`. #### Code samples _Example: cURL · curl_ ``` curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/vision/datasets" ``` ### POST Create a Dataset From a Zip File Asynchronously `POST /v2/vision/datasets/upload` Operation ID: `uploadDatasetAsync_1`. Tags: Vision Prediction. Creates a dataset, labels, and examples from the specified specified .zip file. The call returns immediately and continues to upload data in the background. #### Request body Media type: `multipart/form-data`. Schema: `object`. * `data` (_string_) Path to the .zip file on the local drive (FilePart). * `language` (_string_) Dataset language. Optional. Default is N/A. Reserved for future use. Default: `N/A`. * `name` (_string_) Name of the dataset. Optional. If this parameter is omitted, the dataset name is derived from the file name. Example: `mountainvsbeach`. * `path` (_string_) URL of the .zip file. * `type` (_string_, required) Type of dataset data. Allowed values: `image`, `image-detection`, `image-multi-label`. #### Responses * **200** Upload initiated Media type: `application/json`. Schema: `Dataset`. * **400** Bad request #### Code samples _Example: cURL · curl_ ``` curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "data=@" -F "name=" -F "type=image" "https://api.einstein.ai/v2/vision/datasets/upload" ``` ### POST Create a Dataset From a Zip File Synchronously `POST /v2/vision/datasets/upload/sync` Operation ID: `uploadDatasetSync_1`. Tags: Vision Prediction. Creates a dataset, labels, and examples from the specified .zip file. The call returns after the dataset is created and all of the images are uploaded. #### Request body Media type: `multipart/form-data`. Schema: `object`. * `data` (_string_) Path to the .zip file on the local drive (FilePart). * `language` (_string_) Dataset language. Optional. Default is N/A. Reserved for future use. Default: `N/A`. * `name` (_string_) Name of the dataset. Optional. If this parameter is omitted, the dataset name is derived from the .zip file name. Example: `mountainvsbeach`. * `path` (_string_) URL of the .zip file. * `type` (_string_, required) Type of dataset data. Allowed values: `image`, `image-detection`, `image-multi-label`. #### Responses * **200** Upload success Media type: `application/json`. Schema: `Dataset`. #### Code samples _Example: cURL · curl_ ``` curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "data=@" -F "name=" -F "type=image" "https://api.einstein.ai/v2/vision/datasets/upload/sync" ``` ### GET Get Model Metrics `GET /v2/vision/models/{modelId}` Operation ID: `getVisionTrainedModelMetrics`. Tags: Vision Prediction. Returns the metrics for a model #### Parameters * `modelId` (path, string, required) E.ai Platform Model ID _Example_ ``` JBNHFSDAHN3FIFTEQDKCSCDBIA ``` #### Responses * **200** Model metrics Media type: `application/json`. Schema: `Metrics`. * **404** Resource not found * **502** Unable to complete request due to service error #### Code samples _Example: cURL · curl_ ``` curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/vision/models/" ``` ### DELETE Delete a Model `DELETE /v2/vision/models/{modelId}` Operation ID: `deleteVisionModel`. Tags: Vision Prediction. Deletes the specified model. #### Parameters * `modelId` (path, string, required) E.ai Platform Model ID _Example_ ``` JBNHFSDAHN3FIFTEQDKCSCDBIA ``` #### Responses * **201** Deletion submitted Media type: `application/json`. Schema: `DeletionResponse`. * **404** Resource not found * **502** Unable to complete request due to service error #### Code samples _Example: cURL · curl_ ``` curl -X DELETE -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/vision/models/" ``` ### GET Get Training Status `GET /v2/vision/train/{modelId}` Operation ID: `getVisionTrainStatusAndProgress`. Tags: Vision Prediction. Returns the status of a model's training process. Use the progress field to determine how far the training has progressed. When training completes successfully, the status is SUCCEEDED and the progress is 1. #### Parameters * `modelId` (path, string, required) E.ai Platform modelId _Example_ ``` JBNHFSDAHN3FIFTEQDKCSCDBIA ``` #### Responses * **200** Training status Media type: `application/json`. Schema: `TrainResponse`. * **400** Bad request * **404** Resource not found * **502** Unable to complete request due to service error #### Code samples _Example: cURL · curl_ ``` curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/vision/train/" ``` ### GET Get Model Learning Curve `GET /v2/vision/models/{modelId}/lc` Operation ID: `getVisionTrainedModelLearningCurve`. Tags: Vision Prediction. Returns the metrics for each epoch in a model. #### Parameters * `modelId` (path, string, required) E.ai Platform Model ID _Example_ ``` JBNHFSDAHN3FIFTEQDKCSCDBIA ``` * `offset` (query, integer (int32)) Index of the epoch from which you want to start paging Default: `0`. * `count` (query, integer (int32)) Number of epoch to return. Maximum valid value is 25. Default: `25`. #### Responses * **200** Learning curve Media type: `application/json`. Schema: `LearningCurveList`. * **404** Resource not found * **502** Unable to complete request due to service error #### Code samples _Example: cURL · curl_ ``` curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" "https://api.einstein.ai/v2/vision/models//lc?offset=0&count=25" ``` ### POST Train a Dataset `POST /v2/vision/train` Operation ID: `trainVision`. Tags: Vision Prediction. Trains a dataset and creates a model. #### Request body Media type: `multipart/form-data`. Schema: `object`. * `datasetId` (_integer (int64)_, required) ID of the dataset to train. Example: `57`. * `epochs` (_integer (int32)_) Number of training iterations for the neural network. Optional. Minimum: 1. Maximum: 1000. Example: `20`. * `learningRate` (_number (double)_) N/A when training a detection dataset. * `name` (_string_) Name of the model. Maximum length is 180 characters. Minimum length: 0. Maximum length: 180. * `trainParams` (_string_) JSON that contains parameters that specify how the model is created * `algorithm` (_string_) Algorithm used for train Default: `object-detection-v1`. Example: `object-detection-v1`. #### Responses * **200** Training status Media type: `application/json`. Schema: `TrainResponse`. * **400** Bad request * **404** Resource not found * **502** Unable to complete request due to service error #### Code samples _Example: cURL · curl_ ``` curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "datasetId=" -F "epochs=20" -F "name=" -F "algorithm=object-detection-v1" "https://api.einstein.ai/v2/vision/train" ``` ## Schemas ### ApiUsageList Api Usage List * `object` (_string_) Object returned; in this case, list. Example: `list`. * `data` (_array of ApiUsage_) ### GenerateAccessTokenResponse * `access_token` (_string_) Access token for authorization. Example: `SPFPQ5IBLB6DPE6FKPWHMIWW4MCRICX4M4KQXFQMI6THZXIEZ6QGNWNOERD6S7655LJAFWTRIKC4KGYO5G3XROMEOTBSS53CFSB6GIA`. * `expires_in` (_string_) Number of seconds that the token will expire from the time it was generated. Example: `120`. * `refresh_token` (_string_) Refresh token that can be used to generate an access token. Only returned when you pass the scope=offline parameter to the endpoint. Example: `FL4GSVQS4W5CKSFRVZBLPIVZZJ2K4VIFPLGZ45SJGUQK4SS56IWPWACZ7V2B7OVLVKZCNK5JZSSW7CIHCNQJAO3TOUE3375108HHTLY`. * `token_type` (_string_) Type of token returned. Always Bearer. Example: `Bearer`. ### ELURequest * `text` (_string_, required) text. * `locale` (_string_) (Optional) Input text language Default: `en_US`. * `timeZone` (_string_) (Optional) Timezone for entities normalization Default: `America/Los_Angeles`. * `context` (_Context_) * `agent` (_integer (int32)_) (Optional) Agent/turn identifier for input text (relevant only if historical 'dialogue' field is set) ### ELUResponse * `sentiment` (_SentimentPredictResponse_) * `intents` (_array of IntentPredictionResponse_) Intents responses * `entities` (_EntitiesPredictResponse_) * `polarity` (_PolarityPredictionResponse_) ### EntitiesPredictRequest * `sampleId` (_string_) String that you can pass in to tag the prediction. Optional. Can be any value, and is returned in the response. * `modelId` (_string_) ID of the model that makes the prediction. Default: `ENTITIES`. * `language` (_string_) Language of the document parameter. Default: `en_US`. * `document` (_string_, required) Text to classify. * `timezone` (_string_) Timezone of user. Default: `America/Los_Angeles`. ### EntitiesPredictResponse * `probabilities` (_array of EntitiesResult_) * `sampleId` (_string_) Value passed in when the prediction call was made. Returned only if the sampleId request parameter is provided. Example: `Sample1`. * `object` (_string_) Object returned; in this case, predictresponse. Example: `predictresponse`. ### PredictionErrorResponse * `message` (_string_) * `object` (_string_) ### IntentPredictRequest * `sampleId` (_string_) String that you can pass in to tag the prediction. Optional. Can be any value, and is returned in the response. * `modelId` (_string_, required) ID of the model that makes the prediction. The model must have been created from a dataset with a type of text-sentiment. Example: `WJH4YCA7YX4PCWVNCYNWYHBMY4`. * `numResults` (_integer (int32)_) Number of probabilities to return. Minimum: 1. Example: `3`. * `document` (_string_, required) Text for which you want to return an intent prediction. Example: `I can't tell you how much fun it was`. ### IntentPredictResponse * `probabilities` (_array of LabelResult_) * `sampleId` (_string_) Value passed in when the prediction call was made. Returned only if the sampleId request parameter is provided. Example: `Sample1`. * `object` (_string_) Object returned; in this case, predictresponse. Example: `predictresponse`. ### SentimentPredictRequest * `sampleId` (_string_) String that you can pass in to tag the prediction. Optional. Can be any value, and is returned in the response. * `modelId` (_string_, required) ID of the model that makes the prediction. The model must have been created from a dataset with a type of text-sentiment. Example: `WJH4YCA7YX4PCWVNCYNWYHBMY4`. * `numResults` (_integer (int32)_) Number of probabilities to return. Minimum: 1. Example: `3`. * `document` (_string_, required) Text for which you want to return a sentiment prediction. Example: `I can't tell you how much fun it was`. ### SentimentPredictResponse * `probabilities` (_array of LabelResult_) * `sampleId` (_string_) Value passed in when the prediction call was made. Returned only if the sampleId request parameter is provided. Example: `Sample1`. * `object` (_string_) Object returned; in this case, predictresponse. Example: `predictresponse`. ### Dataset * `id` (_integer (int64)_) Dataset ID. Example: `1000014`. * `name` (_string_) Name of the dataset. Example: `weather report`. * `createdAt` (_string (date-time)_) Date and time that the dataset was created. * `updatedAt` (_string (date-time)_) Date and time that the dataset was last updated. * `labelSummary` (_LabelSummary_) * `totalExamples` (_integer (int32)_) Total number of examples in the dataset. Example: `20`. * `totalLabels` (_integer (int32)_) Total number of labels in the dataset. Example: `2`. * `available` (_boolean_) Specifies whether the dataset is ready to be trained. * `statusMsg` (_string_) Status of the dataset creation and data upload. Example: `SUCCEEDED`. * `type` (_string_) Type of dataset data. * `language` (_string_) Dataset language. Example: `N/A`. * `numOfDuplicates` (_integer (int32)_) Number of duplicate images. This number includes duplicates in the .zip file from which the dataset was created plus the number of duplicate images from subsequent PUT calls to add images to the dataset. * `entityName` (_string_) * `filter` (_string_) * `changesetId` (_integer (int32)_) Changeset ID. Example: `1`. * `object` (_string_) Object returned; in this case, dataset. Example: `dataset`. ### DeletionResponse * `deletedObjectId` (_string_) ID of the object deleted. Depending on the object you delete, this contains the dataset ID or the model ID. Example: `1003360`. * `id` (_string_) ID of the deletion Example: `Z2JTFBF3A7XKIJC5QEJXMO4HSY`. * `message` (_string_) Additional information about the deletion. For example, a message is returned if the deletion fails. * `organizationId` (_string_) ID of the org to which the dataset or model being deleted belongs. Example: `2`. * `progress` (_number (double)_) How far the deletion has progressed. Values are between 0–1. Example: `1`. * `status` (_string_) Status of the deletion. Allowed values: `QUEUED`, `RUNNING`, `SUCCEEDED_WAITING_FOR_CACHE_REMOVAL`, `SUCCEEDED`, `KILLED`, `FAILED`, `RETRY`. * `type` (_string_) Object that's being deleted Allowed values: `DATASET`, `MODEL`. * `object` (_string_) Object returned; in this case, deletion. Example: `deletion`. ### ExampleList * `object` (_string_) Object returned; in this case, list. Example: `list`. * `data` (_array of Example_) ### ModelList * `object` (_string_) Object returned; in this case, list. Example: `list`. * `data` (_array of TrainedModelResponse_) ### DatasetList * `object` (_string_) Object returned; in this case, list. Example: `list`. * `data` (_array of Dataset_) ### Metrics * `id` (_string_) Model Id * `metricsData` (_string_) Model metrics values. * `createdAt` (_string (date-time)_) Date and time that the model was created. * `language` (_string_) * `algorithm` (_string_) * `object` (_string_) ### TrainResponse * `datasetId` (_integer (int64)_) ID of the dataset trained to create the model. Example: `57`. * `datasetVersionId` (_integer (int64)_) Not available yet Example: `0`. * `name` (_string_) Name of the model. Example: `My Model - Version1`. * `status` (_string_) Status of the model. Allowed values: `QUEUED`, `RUNNING`, `SUCCEEDED`, `FAILED`, `KILLED`, `FAILED_WITH_RETRIES`. * `progress` (_number_) How far the dataset training has progressed. Values are between 0–1. Example: `0.7`. * `createdAt` (_string (date-time)_) Date and time that the model was created. * `updatedAt` (_string (date-time)_) Date and time that the model was last updated. * `modelId` (_string_) ID of the model. Contains letters and numbers. Example: `2KXJEOM3N562JBT4P7OX7VID2Q`. * `hawkingModelArtifactId` (_string_) * `failureMsg` (_string_) Reason the dataset training failed. Returned only if the training status is FAILED. Example: `To train a dataset and create a model, the dataset must contain at least 100 examples per label for test set`. * `learningRate` (_number (double)_) Learning rate used during training. Example: `0.0001`. * `epochs` (_integer (int32)_) Number of epochs used during training. Example: `20`. * `queuePosition` (_integer (int32)_) Where the training job is in the queue. This field appears in the response only if the status is QUEUED. Example: `1`. * `modelType` (_string_) Type of data from which the model was created. * `algorithm` (_string_) Algorithm used to create the model. Returned only when the modelType is image-detection. Example: `object-detection`. * `language` (_string_) Model language inherited from the dataset language. For image datasets, default is N/A. For text datasets, default is en\_US. Example: `en_US`. * `object` (_string_) Object returned; in this case, training. Example: `training`. * `trainParams` (_string_) Training parameters passed into the request. Example: `{"trainSplitRatio":0.7}`. * `trainStats` (_string_) Returns null when you train a dataset. Training statistics are returned when the status is SUCCEEDED or FAILED. ### LearningCurveList * `object` (_string_) Object returned; in this case, list. Example: `list`. * `data` (_array of LearningCurve_) ### OCRPredictResponse * `object` (_string_) Object returned; in this case, predictresponse. Example: `predictresponse`. * `probabilities` (_array of OCRResult_) * `task` (_string_) Same value as request parameter. Returns text if the request parameter isn't supplied. Example: `Task1`. * `sampleId` (_string_) Same value as request parameter. Returned only if the sampleId request parameter is provided. Example: `Sample1`. ### ApiUsage * `id` (_integer (int64)_, required) Id Example: `489`. * `organizationId` (_string_, required) Organization Id Example: `108`. * `type` (_string_, required) Type * `count` (_integer (int64)_, required) Count * `planData` (_array of PlanData_, required) Plan Data as Json * `startsAt` (_string (date-time)_, required) Starts At Example: `2017-03-01T00:00:00Z`. * `endsAt` (_string (date-time)_, required) Ends At Example: `2017-04-01T00:00:00Z`. * `licenseId` (_string_, required) License ID Example: `kJCHtYDCSf`. * `predictionsRemaining` (_integer (int32)_, required) Predictions Remaining Example: `1997`. * `predictionsUsed` (_integer (int32)_, required) Predictions Used Example: `3`. * `predictionsMax` (_integer (int32)_, required) Predictions Max Example: `2000`. * `object` (_string_, required) object Example: `apiusage`. ### Context (Optional) Context * `models` (_array of ELUModel_) List of models to be used Minimum items: 0. Maximum items: 3. * `dialogue` (_array of Utterance_) (Optional) History of the current conversation Minimum items: 0. Maximum items: 5. ### IntentPredictionResponse Intents responses * `probabilities` (_array of LabelResult_) * `sampleId` (_string_) Value passed in when the prediction call was made. Returned only if the sampleId request parameter is provided. Example: `Sample1`. * `object` (_string_) Object returned; in this case, predictresponse. Example: `predictresponse`. * `model` (_ELUModel_) ### PolarityPredictionResponse Polarity responses * `probabilities` (_array of LabelResult_) ### EntitiesResult Array of probabilities for the prediction. * `token` (_string_, required) Characters from the document parameter to which the returned label applies * `label` (_string_, required) Probability label for the input * `probability` (_number (float)_, required) Probability value for the input. Values are between 0–1 * `span` (_Span_, required) * `normalizedData` (_array of NormalizedValue_, required) Normalization values for the entity. The format differs based on the normalization type (one of person, range or string) ### LabelResult label * `probability` (_number (float)_) Probability value for the input. Values are between 0–1. * `label` (_string_) Probability label for the input. ### LabelSummary Contains the labels array that contains all the labels for the dataset. * `labels` (_array of Label_) ### Example * `id` (_integer (int64)_, required) ID of the example. Example: `546`. * `name` (_string_, required) Name of the example. Example: `659803277.jpg`. * `location` (_string_) Location of the image in the dataset. Example: `1798f7e1f628a22f450c50c17247b314b392296758581ec688ea418b8a413df71.jpg`. * `createdAt` (_string (date-time)_) Date and time that the example was created. * `label` (_Label_) * `object` (_string_) Object returned; in this case, example. Example: `example`. ### TrainedModelResponse * `object` (_string_) Object returned; in this case, model. Example: `model`. * `modelId` (_string_) E.ai Model ID. Example: `13JK34JJ44`. * `hawkingModelId` (_string_) Hawking model ID. Example: `sdfas324234`. * `hawkingPipelineArtifactId` (_string_) Hawking pipeline artifact ID. Example: `sdfjksdajflkasdjf`. * `createdAt` (_string (date-time)_) Date and time that the model was created. * `updatedAt` (_string (date-time)_) Date and time that the model was last updated. * `datasetId` (_integer (int64)_) Eai dataset ID Example: `6000059`. * `hawkingTrainingFlowId` (_string_) Hawking training flow ID Example: `5923c823-e7f1-4f1d-b3c6-61450cae697a`. * `hawkingPipelineRunId` (_string_) Hawking pipeline run ID Example: `5923c823-e7f1-4f1d-b3c6-61450cae697a_1`. * `name` (_string_) Name of the model Example: `Weather Intent Model`. * `status` (_string_) status of model Allowed values: `QUEUED`, `RUNNING`, `SUCCEEDED`, `FAILED`, `KILLED`, `FAILED_WITH_RETRIES`. Example: `QUEUED`. * `progress` (_number_) progress of model Example: `0.7`. ### LearningCurve * `object` (_string_) Object returned; in this case, learningcurve. Example: `learningcurve`. * `epoch` (_object_) Epoch to which the metrics correspond. Example: `1`. * `metricsData` (_object_) Model metrics values. * `epochResults` (_object_) Prediction results for the set of data used to test the model during training. ### OCRResult Array of probabilities for the prediction. * `probability` (_number (float)_) Probability value for the input. Values are between 0–1. * `label` (_string_) Content of the detected text. * `boundingBox` (_BoundingBox_) * `attributes` (_Attributes_) ### PlanData Plan Data as Json * `plan` (_string_) Example: `STARTER`. * `amount` (_integer (int32)_) Example: `1`. * `source` (_string_) Example: `HEROKU`. ### ELUModel Models definition to consume for Insights * `id` (_string_, required) id * `type` (_string_, required) type Allowed values: `intents`, `entities`, `co-references`, `negation`, `sentiment`, `polarity`, `slots`. * `numResults` (_integer (int32)_) numResults ### Utterance One turn of a conversation * `agent` (_integer (int32)_, required) Agent id * `text` (_string_, required) Input document ### Span Contains the starting character position and ending character position of the token with the document string * `end` (_integer (int32)_, required) end * `start` (_integer (int32)_, required) start ### NormalizedValue Normalization values for the entity. The format differs based on the normalization type (one of person, range or string) * `firstName` (_string_) Person's first name if detected. * `lastName` (_string_) Person's last name if detected. * `middleName` (_string_) Person's middle name if detected. * `salutation` (_string_) Person's salutation if detected. * `suffix` (_string_) Person's suffix if detected. * `from` (_NormalizedString_) * `to` (_NormalizedString_) * `unit` (_string_) Unit of measure. Can be empty. * `value` (_string_) Entity value normalized to a standard format. * `issuer` (_string_) credit card issuer * `type` (_string_) Type of normalization. Use this to interpret the normalized data ### Label * `id` (_integer (int64)_) ID of the label. Example: `621`. * `datasetId` (_integer (int64)_, required) ID of the dataset that the label belongs to. Example: `57`. * `name` (_string_, required) Name of the label. Example: `Mountain`. * `numExamples` (_integer (int64)_) Number of examples that have the label. Example: `40`. ### BoundingBox * `x1` (_integer (int32)_, write-only) * `y1` (_integer (int32)_, write-only) * `x2` (_integer (int32)_, write-only) * `y2` (_integer (int32)_, write-only) * `minX` (_integer (int32)_) X-coordinate of the right side of the bounding box. Number of pixels from the left edge of the image. * `minY` (_integer (int32)_) Y-coordinate of the bottom of the bounding box. Number of pixels from the top edge of the image. * `maxX` (_integer (int32)_) X-coordinate of the left side of the bounding box. The origin of the coordinate system is the top-left of the image. Number of pixels from the left edge of the image. * `maxY` (_integer (int32)_) Y-coordinate of the top of the bounding box. Number of pixels from the top edge of the image. ### Attributes Contains additional attributes related to the task parameter. If the task parameter is table, the row and column IDs for the detected text are returned. If the task parameter is contact, the detected entity tags will be returned. * `cell` (_CellLocation_) * `tag` (_string_) * `language` (_string_) * `value` (_EntityObject_) * `pageBlockType` (_string_, write-only) * `pageObjectUnit` (_array of PageObjectUnit_, write-only) * `pageNumber` (_string_) * `tableId` (_string_) * `cellLocation` (_CellLocation_) ### NormalizedString Type of normalization. Use this to interpret the normalized data * `unit` (_string_) Unit of measure. Can be empty. * `value` (_string_) Entity value normalized to a standard format. * `type` (_string_) type ### CellLocation * `rowIndex` (_integer (int32)_) Index of the row that contains the detected text. * `colIndex` (_integer (int32)_) Index of the column that contains the detected text. ### EntityObject * `text` (_string_) * `entity` (_string_) * `boundingBox` (_BoundingBox_) * `normalizedText` (_string_) ### PageObjectUnit * `text` (_string_) * `boundingBox` (_BoundingBox_) * `pageObjectUnitAttributes` (_PageObjectUnitAttributes_) ### PageObjectUnitAttributes * `bold` (_boolean_) * `cell` (_CellLocation_) * `cellLocation` (_CellLocation_) # Entity Normalization The Einstein NER (beta) model returns a prediction with normalized values for all entities except for LOCATION, ORGANIZATION, and PERSON. What does it mean that the model normalizes certain types of entities? It means that when you send text in for prediction and the model identifies an entity that supports normalization, the API returns the actual value for the entity and other information. For example, if today is March 21, 2020, and your text contains the word yesterday, the model returns a normalized value of `2020-03-20 00:00:00`. Here are some examples of how various date, time, and percent values returned in the prediction are normalized. We also provide limited support for compound fields (address information only), as the final example in this table shows. | Token | Normalized value | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | early January, 1993 | 1993-01-01T00:00:00.000-08:00 | | 3 days from now | 2020-08-17T08:00:00.000-07:00 | | 3 weeks ago | 2020-07-24T00:00:00.000-07:00 | | twenty five minutes ago | 2020-08-14T08:29:43.426-07:00 | | on Thursday | 2020-08-20T00:00:00.000-07:00 | | 22% | 0.22 | | 123 Main Street, Suite 54, San Francisco, CA, 94101, U.S.A. | {"Street":"123 Main Street Suite 54","City":"San Francisco","State":"CA","ZipCode":"94101","Country":"U.S.A"}

**Note** Empty fields are excluded from the compound JSON response. | See the [Entity Reference](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/entity-reference.md) for more information about the normalized data for each entity. # Entity Reference The Einstein NER (beta) API supports normalization for some entities. Normalized data is returned in the `normalizedData` array. The type and format of the data depends on the type of entity. ## CREDIT CARD **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | -------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | | `type` | string | Normalization type. Valid values:
  • `credit_card`
| 2.0 | | `issuer` | string | Credit card issuer. Valid values:
  • American Express
  • Diner's Club
  • Discover
  • Japan Credit Bureau
  • Maestro Card
  • Mastercard
  • Visa
  • other
| 2.0 | | `value` | string | Value of the normalized data | 2.0 | With the exception of the _ja_ locale, credit card numbers are detected regardless of the locale provided in the request. When the request document is passed with _ja_ locale, we support numbers written in full-width characters. **Examples** ```json "probability": 1.0, "label": "CREDIT-CARD-NUMBER", "token": "5555-5555-5555-4444", "span": { "start": 0, "end": 19 }, "normalizedData": [ { "value": "5555555555554444", "issuer": "Mastercard", # credit card issuer "type": "credit_card" } ] ``` **note:** Though we identify valid credit card number formats (using Luhn’s algorithm test), we do not post process the value to determine whether a number is a real-world credit card. ## DATETIME **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | ------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | | `type` | string | Normalization type. Valid values:
  • `value`—Object contains the normalized value for the token.
  • `range`—Object contains from and to objects that represent the start value and end value for the token.
| 2.0 | | `unit` | string | Unit of measurement for the token value. Valid values:
  • `day`
  • `hour`
  • `minute`
  • `month`
  • `quarter`
  • `second``week`
  • `year`

The unit returned by the API is determined by the unit in the token. For example, if the token is "between today and tomorrow" the unit is day. | 2.0 | | `value` | string | Value of the normalized data. | 2.0 | This entity can sometimes overlap with the DURATION and NUMBER entities. **Examples** ```json { "token": "...", "label": "DATETIME", "probability": 0.9787207, "normalizedData": [ { "type": "value", "unit": "hour", "value": "2020-02-03T17:00:00.000-08:00" // 5pm }, ... or ... { "type": "range", // between 4pm and 5pm "from": { "type": "value", "unit": "hour", "value": "2020-02-03T16:00:00.000-08:00" }, "to": { "type": "value", "unit": "hour", "value": "2020-02-03T17:00:00.000-08:00" } } ] } ``` ```json { "probabilities": [ { "probability": 0.84189475, "label": "DATETIME", "token": "between today and tomorrow", "span": { "start": 22, "end": 48 }, "normalizedData": [ { "from": { "value": "2020-08-10T00:00:00.000-07:00", // Today's date "unit": "day", "type": "value" }, "to": { "value": "2020-08-12T00:00:00.000-07:00", // Tomorrow's date "unit": "day", "type": "value" }, "type": "range" } ] } ], "object": "predictresponse" } ``` ## DURATION **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | ------- | ------ | ------------------------------------------------------------------------------------------------------------------- | ----------------- | | `type` | string | Normalization type. Valid values:
  • `value`—Object contains the normalized value for the token.
| 2.0 | | `unit` | string | Unit of measurement for the token value. Valid values:
  • `second`—Number of seconds in the duration.
| 2.0 | | `value` | string | Value of the normalized data. | 2.0 | This entity can sometimes overlap with the DATETIME and NUMBER entities. **Example** ```json { "token": "3 hours", "label": "DURATION", "probability": 0.9125207, "normalizedData": [ { "type": "value", "value": "10800", "unit": "second" } ] } ``` ## EMAIL **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | ------- | ------ | -------------------------------------------------------------------------------------------------------------- | ----------------- | | `type` | string | Normalization type. Valid values:
  • `value`—Object contains the normalized value for the token.
| 2.0 | | `unit` | string | Empty for the EMAIL entity type. | 2.0 | | `value` | string | Value of the normalized data. | 2.0 | The API recognizes email addresses in these two formats. - info at salesforce dot com - info@salesforce.com **Example** ```json { "probabilities": [ { "probability": 1, "label": "EMAIL", "token": "info at salesforce dot com", "span": { "start": 0, "end": 26 }, "normalizedData": [ { "value": "info@salesforce.com", "unit": "", "type": "value" } ] } ], "object": "predictresponse" } ``` ## LOCATION **Supports normalization:** No **Example** ```json { "token": "London", "label": "LOCATION", "probability": 0.9787207, "normalizedData": [], "span": { ... } } ``` ## MONEY **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | ------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | | `type` | string | Normalization type. Valid values:
  • `value`—Object contains the normalized value for the token.
| 2.0 | | `unit` | string | Unit of measurement for the token value. Valid values:
  • `EUR`—Euro
  • `GBP`—British pound stering
  • `RUB`—Russian ruble
  • `USD`—United States dollar
| 2.0 | | `value` | string | Value of the normalized data. For money, it's the numeric value of the specified currency. | 2.0 | **Example** ```json { "probabilities": [ { "probability": 0.95074475, "label": "MONEY", "token": "five dollars", "span": { "start": 21, "end": 33 }, "normalizedData": [ { "value": "5", "unit": "USD", "type": "value" } ] } ], "object": "predictresponse" } ``` ## NUMBER **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | ------- | ------ | -------------------------------------------------------------------------------------------------------------- | ----------------- | | `type` | string | Normalization type. Valid values:
  • `value`—Object contains the normalized value for the token.
| 2.0 | | `unit` | string | Empty for the NUMBER entity type. | 2.0 | | `value` | string | Value of the normalized data. | 2.0 | **Example** ```json { "probabilities": [ { "probability": 1, "label": "NUMBER", "token": "Three thousand", "span": { "start": 0, "end": 14 }, "normalizedData": [ { "value": "3000", "unit": "", "type": "value" } ] } ], "object": "predictresponse" } ``` ## ORGANIZATION **Supports normalization:** No **Example** ```json { "probability": 0.9469566, "label": "ORGANIZATION", "token": "Salesforce", "span": { "start": 25, "end": 35 }, "normalizedData": [] } ``` ## PERCENT **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | ------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | | `type` | string | Normalization type. Valid values:
  • `value`—Object contains the normalized value for the token.
  • `range`—Object contains from and to objects that represent the start value and end value for the token.
| 2.0 | | `unit` | string | Empty for the PERCENT entity type. | 2.0 | | `value` | string | Value of the normalized data. | 2.0 | The normalized value is the percent divided by 100. For example, 50 percent is 0.5. Recognizes percentages in these formats: - 50% - 50 percent **Example** ```json // Single value example { "token": "150%", "label": "PERCENT", "probability": 0.9787207, "normalizedData": [ { "type": "value", "unit": "", "value": "1.5" } ] } // Range value example { "probability": 0.99382097, "label": "PERCENT", "token": "40-50%", "span": { "start": 26, "end": 32 }, "normalizedData": [ { "from": { "value": "0.4", "unit": "", "type": "value" }, "to": { "value": "0.5", "unit": "", "type": "value" }, "type": "range" } ] } ``` ## PERSON **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | ------------ | ------ | ------------------------------------------------------------------- | ----------------- | | `firstName` | string | Text identified as the person's first name. | 2.0 | | `lastName` | string | Text identified as the person's last name. | 2.0 | | `middleName` | string | Text identified as the person's middle name. | 2.0 | | `salutation` | string | Empty for the PERSON entity type. | 2.0 | | `suffix` | string | Text identified as the person's suffix such as Jr., Sr., and so on. | 2.0 | | `type` | string | Normalization type. Valid values:
  • `person`
| 2.0 | **note:** Titles are not detected. For example, for the string _Dr. Jim Blimpish_ only _Jim_ and _Blimpish_ are detected. **Example** ```json "probability": 0.99966216, "label": "PERSON", "token": "Marc Benioff", "span": { "start": 0, "end": 12 }, "normalizedData": [ { "firstName": "Marc", "lastName": "Benioff", "middleName": "", "suffix": "", "salutation": "", "type": "person" } ] ``` ## PHONE-NUMBER **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | ------- | ------ | -------------------------------------------------------------------------------------------------------------- | ----------------- | | `type` | string | Normalization type. Valid values:
  • `value`—Object contains the normalized value for the token.
| 2.0 | | `unit` | string | Empty for the PHONE-NUMBER entity type. | 2.0 | | `value` | string | Value of the normalized data. | 2.0 | **Example** ```json { "probability": 1, "label": "PHONE-NUMBER", "token": "408-123-4455", "span": { "start": 37, "end": 49 }, "normalizedData": [ { "value": "4081234455", "unit": "", "type": "value" } ] } ``` ## URL **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | ------- | ------ | -------------------------------------------------------------------------------------------------------------- | ----------------- | | `type` | string | Normalization type. Valid values:
  • `value`—Object contains the normalized value for the token.
| 2.0 | | `unit` | string | Empty for the URL entity type. | 2.0 | | `value` | string | Value of the normalized data. | 2.0 | ```json { "probabilities": [ { "probability": 1, "label": "URL", "token": "https://developer.salesforce.com/signup.", "span": { "start": 58, "end": 98 }, "normalizedData": [ { "value": "https://developer.salesforce.com/signup.", "unit": "", "type": "value" } ] } ], "object": "predictresponse" } ``` ## SOCIAL SECURITY NUMBER **Supports normalization:** Yes **Normalization fields** | Name | Type | Description | Available Version | | ------- | ------ | -------------------------------------------------------------------- | ----------------- | | `type` | string | Normalization type. Valid values:
  • `social_security`
| 2.0 | | `value` | string | Value of the normalized data. | 2.0 | **EXAMPLE** ```json "probability": 1.0, "label": "SOCIAL-SECURITY-NUMBER", "token": "135-24-2244", "span": { "start": 0, "end": 11 }, "normalizedData": [ { "value": "135242244", "type": "social_security" } ] ``` **note:** Though we identify valid social security number formats we do not post process the value to determine whether a number is a real-world social security number. # Extract Data from Custom Forms (Beta) ## Extract Entity Value Pairs The custom form solution allows the users to specify what information they want to extract from their custom forms. The expected entity names are defined in a form template in JSON format, and for each entity, the user can provide its key variations to the best of one’s knowledge. The key variations are not necessarily a complete list, but the more variations are provided, the better our ML model can understand the concept of this entity. For instance, if the user wants to extract the date of birth from forms, he/she can define an entity named `date of birth`, and provide its key variations in a list like `["date of birth", "birthday", "DOB", "birth date"]`. ## Extract Tables The custom form solution automatically detects all the tables, recognize the texts in cells and infer the location, row and column number of each table cell. Set `AutomaticallyRecognizeTables` in the form template to "true" to enable this function. Here's an example of a form template that extracts the date of birth and last name, and detect tables. ```text { "Fields": [ { "key": { "entity": "date_of_birth", "entity_type": [], "text": [ "date of birth", "birthday", "DOB", "birth date" ] } }, { "key": { "entity": "last_name", "entity_type": [], "text": [ "last name", "family name", "surname" ] } } ], "Tables": [], "AutomaticallyRecognizeTables": true, "Version": "3.0" } ``` When you call the API, send in the form as an image or PDF, set `task` to `form`, set `formTemplate` to the path of form template JSON file, and specify the `tabulatev2` modelId. The JSON response contains entity-value pairs for each field in the form. # Extract Data from Invoices (Beta) The Invoice solution extracts important information such as invoice number, invoice date, due date, total amount, etc., from any invoice image or PDF. In real world, there are many key word variations for the same concept. For instance, “invoice #”, “invoice number”, “INV No.”, etc., all refer to the same thing. Our solution leverages machine learning models to understand the document context, consolidates the key variations in the same entity (for example, `invoice_number` ) and extracts the correct values accordingly. The Einstein OCR invoice solution currently supports the list of entities below. | Entity Name | Key Variation Examples | | ------------------ | ------------------------------------------------- | | `invoice_number` | invoice number, invoice #, invoice, invoice ID... | | `invoice_date` | invoice date, date, ... | | `due_date` | due date, due on, ... | | `purchase_order` | PO number, PO#, purchase order, ... | | `total_amount` | total, total amount, ... | | `total_tax_amount` | tax, total tax, ... | | `amount_due` | due, amount due, ... | When you call the API, send in the form as an image or PDF, set `task` to `invoice` and specify the `tabulatev2` modelId. The JSON response contains entity-value pairs for each field in the form. ![ExtractDataFromInvoicesExample](https://developer.salesforce.com/media/Extract_Data_From_Invoices.png) In the example above, the extracted entity value pairs are: | Entity | Value | | ---------------- | --------- | | `invoice_number` | 940226 | | `invoice_date` | 2/18/1994 | | `due_date` | 3/20/1994 | | `total_amount` | 852.8 | # Form 1040 This table lists the entities returned by the OCR model for Form 1040. Supports only the 2019 Form 1040. ## Page 1 | Entity name | Form key | | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `year` | | | `first_name_and_initial` | Your first name and middle initial | | `last_name` | Last name | | `social_security_number` | Your social security number | | `spouse_first_name` | If joint return, spouse’s first name and middle initial | | `spouse_last_name` | Last name | | `spouse_social_security_number` | Spouse’s social security number | | `address_number_and_street` | Home address (number and street). If you have a P.O. box, see instructions. | | `address_apt_number` | Apt no. | | `address_city_state_zip` | City, town or post office, state, and ZIP code. If you have a foreign address, also complete spaces below (see instructions). | | `foreign_country` | Foreign country name | | `foreign_state` | Foreign province/state/county | | `foreign_postal_code` | Foreign postal code | | `1_wages_salaries_tips` | 1 | | `2a_tax_exempt_interest` | 2a | | `2b_taxable_interest` | 2b | | `3a_qualified_dividends` | 3a | | `3b_ordinary_dividends` | 3b | | `4a_ira_distributions` | 4a | | `4b_taxable_amount` | 4b | | `4c_pensions_and_annuities` | 4c | | `4d_taxable_amount` | 4d | | `5a_social_security_benefits` | 5a | | `5b_taxable_amount` | 5b | | `6_capital_gain_or_loss` | 6 | | `7a_other_income_from_schedule_1_line_9` | 7a | | `7b_total_income` | 7b | | `8a_adjustments_to_income_from_schedule_1_line_22` | 8a | | `8b_adjusted_gross_income` | 8b | | `9_standard_deduction_or_itemized_deductions` | 9 | | `10_qualified_business_income_deduction` | 10 | | `11a_add_lines_9_and_10` | 11a | | `11b_taxable_income` | 11b | ## Page 2 | Entity name | Form key | | ------------------------------------------------------------------ | --------------------------------------------------------------------------------- | | `12a_tax` | 12a | | `12b_add_schedule_2_line_3_and_line_12a_and_enter_the_total` | 12b | | `13a_child_tax_credit_or_credit_for_other_dependents` | 13a | | `13b_add_schedule_3_line_7_and_line_13a_and_enter_the_total` | 13b | | `14_subtract_line_13b_from_line_12b` | 14 | | `15_other_taxes` | 15 | | `16_total_tax` | 16 | | `17_federal_income_tax_withheld_from_forms_w2_and_1099` | 17 | | `18a_earned_income_credit` | 18a | | `18b_additional_child_tax_credit` | 18b | | `18c_american_opportunity_credit` | 18c | | `18d_schedule_3_line_14` | 18d | | `18e_total_other_payments_and_refundable_credits` | 18e | | `19_total_payments` | 19 | | `20_overpaid` | 20 | | `21a_amount_of_line_20_you_want_refunded_to_you` | 21a | | `routing_number` | Routing number | | `account_number` | Account number | | `22_amount_of_line_20_you_want_applied_to_your_2020_estimated_tax` | 22 | | `23_amount_you_owe` | 23 | | `24_estimated_tax_penalty` | 24 | | `designee_name` | Designee’s name | | `phone_number` | Phone no. | | `designee_pin` | Personal identification number (PIN) | | `signature` | Your signature | | `date` | Date | | `occupation` | Your occupation | | `pin` | If the IRS sent you an Identity Protection PIN, enter it here (see inst.) | | `spouse_signature` | Spouse’s signature. If a joint return, both must sign. | | `spouse_date` | Date | | `spouse_occupation` | Spouse’s occupation | | `spouse_pin` | If the IRS sent your spouse an Identity Protection PIN, enter it here (see inst.) | | `phone_number` | Phone no. | | `email` | Email address | | `preparer_name` | Preparer’s name | | `preparer_signature` | Preparer’s signature | | `preparer_signature` | Date | | `ptin` | PTIN | | `firm_name` | Firm’s name | | `firm_phone_number` | Phone no. | | `firm_address` | Firm’s address | | `firm_ein` | Firm’s EIN | # Form W-2 This table lists the entities returned by the OCR model for Form W-2. | Entity name | Form key | | ------------------------------------- | ---------------------------------------- | | `a_employee_social_security_number` | a Employee's social security number | | `b_employer_identification_number` | b Employer identification number (EIN) | | `c_employer_name_address_and_zipcode` | c Employer's name, address, and ZIP code | | `d_control_number` | d Control number | | `e_employee_first_name_and_initial` | e Employee’s first name and initial | | `last_name` | Last name | | `suff` | Suff. | | `f_employee_address_and_zipcode` | f Employee’s address and ZIP code | | `1_wages_tips_other_compensation` | 1 Wages, tips, other compensation | | `2_federal_income_tax_withheld` | `2 Federal income tax withheld` | | `3_social_security_wages` | 3 Social security wages | | `4_social_security_tax_withheld` | 4 Social security tax withheld | | `5_medicare_wages_and_tips` | 5 Medicare wages and tips | | `6_medicare_tax_withheld` | 6 Medicare tax withheld | | `7_social_security_tips` | 7 Social security tips | | `8_allocated_tips` | 8 Allocated tips | | `9_verification_code` | 9 Verification code | | `10_dependent_care_benefits` | 10 Dependent care benefits | | `11_nonqualified_plans` | 11 Nonqualified plans | | `12a_see_instructions_for_box_12` | 12a See instructions for box 12 | | `12b` | 12b | | `14_other` | 14 Other | | `12c` | 12c | | `12d` | 12d | | `15_state` | 15 State | | `employer_state_id_number` | Employer's state ID number | | `16_state_wages_tips_etc` | 16 State wages, tips, etc. | | `17_state_income_tax` | 17 State income tax | | `18_local_wages_tips_etc` | 18 Local wages, tips, etc. | | `19_local_income_tax` | 19 Local income tax | | `20_locality_name` | 20 Locality name | # Generate an OAuth Access Token A JWT access token is required to make any Einstein Vision and Language API calls. There are three ways to generate an access token. - Web UI—Use the [token page](https://api.einstein.ai/token) to enter your username (most likely your email address), upload your private key file, and generate a JWT token. - Programmatically using your key—Load your private key, generate an assertion, and call the API to get an access token. You must monitor when the token expires and generate a new one. - Programmatically using a refresh token—Load your private key, generate an assertion, and call the API to get a refresh token. Use that refresh token from then on call the API and generate an access token. # Generate an OAuth Token Using a Refresh Token If you don't want to generate an access token using your private key, you can use a refresh token. A refresh token is a JWT token that never expires. You can use a refresh token only to generate an access token; you can't use it to make an authenticated API call. This is useful in cases where the client making API calls doesn't have access to the private key. A third-party system can generate the refresh token and provide it to the client making API calls. **warning:** A refresh token never expires and is used to generate access tokens used to make API calls. Be sure to safeguard refresh tokens the same way you would any password. To get an access token using a refresh token, you must first get the refresh token. Then you use the refresh token from then on to generate an access token. ## Generate a Refresh Token 1. Open the `einstein_platform.pem` file and read in the key contents. 2. Create the JWT payload. The payload is JSON that contains: - `sub`—Your Einstein Vision and Language username. You can find your username in the welcome email you receive after you get an account. If you signed up using Salesforce, your username is the email address associated with the org you signed up with. - `aud`—The API endpoint URL for generating a token. - `exp`—The expiration time in Unix time. This value is the current Unix time in seconds plus the number of seconds you want the token to be valid. For testing purposes, you can get the Unix time at [Time.is](https://time.is/Unix_time_now). The JWT payload looks like this JSON. ```json { "sub": "", "aud": "https://api.einstein.ai/v2/oauth2/token", "exp": } ``` 3. Sign the JWT payload with your RSA private key to generate an assertion. The private key is contained in the `einstein_platform.pem` file you downloaded when you signed up for an account. The code to generate the assertion varies depending on your programming language. If you're doing manual testing, you can generate an assertion using [jwt.io](https://jwt.io/). 4. Call the API and pass in the assertion along with the `scope=offline` parameter. You pass in all the necessary data in the `-d` parameter. Replace `` with the assertion you generated. This cURL command shows the call. ```cURL curl -H "Content-type: application/x-www-form-urlencoded" -X POST https://api.einstein.ai/v2/oauth2/token -d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=&scope=offline" ``` The response looks similar to this JSON. ```json { "access_token": "SPFPQ5IBLB6DPE6FKPWHMIWW4MCRICX4M4KQXFQMI6THZXIEZ6QGNWNOERD6S7655LJAFWTRIKC4KGYO5G3XROMEOTBSS53CFSB6GIA", "refresh_token": "FL4GSVQS4W5CKSFRVZBLPIVZZJ2K4VIFPLGZ45SJGUQK4SS56IWPWACZ7V2B7OVLVKZCNK5JZSSW7CIHCNQJAO3TOUE3375108HHTLY", "token_type": "Bearer", "expires_in": "120" } ``` 5. Store the refresh token. ## Get an Access Token Using the Refresh Token Now that you have a refresh token, you can use it to generate an access token that you can use to call the API. 1. Call the `/v2/oauth2/token` endpoint and pass the refresh token along with these parameters. - `grant_type`—Specify the string `refresh_token`. - `refresh_token`—The refresh token you created. - `valid_for`—Number of seconds until the access token expires. Default is 60 seconds. Maximum value is 30 days (2592000 seconds). ```cURL curl -H "Content-type: application/x-www-form-urlencoded" -X POST https://api.einstein.ai/v2/oauth2/token -d "grant_type=refresh_token&refresh_token=FL4GSVQS4W5CKSFRVZBLPIVZZJ2K4VIFPLGZ45SJGUQK4SS56IWPWACZ7V2B7OVLVKZCNK5JZSSW7CIHCNQJAO3TOUE3375108HHTLY&valid_for=60" ``` The response looks like this JSON. ```json { "access_token": "LA5JPHC6J2FOVPXVU36HW7WUF3GNNZC5PINC6NX272CEFCWGQNIBB3TSVEAUHD6SYZFED27YRDMRQDIEUNNRT7HXFJTNJCFU5DXNNOI", "token_type": "Bearer", "expires_in": "60" } ``` Use the access token to make any authenticated API calls as long as the token is valid (it's not expired). When the access token expires, use the refresh token to generate a new one. You can't use refresh token to generate another refresh token. The `scope=offline` parameter doesn't work for this call. To delete a token, use the call to [Delete a Refresh Token](/docs/analytics/einstein-vision-language/references). # Generate an OAuth Token Using Your Key When you sign up for an account, you download your key contained in a file called einstein_platform.pem. You can quickly get a token using the [token page](https://api.einstein.ai/web/token) and uploading your key. In code, however, you must programmatically get an OAuth token using your key. You do this by generating an assertion and then passing that to the API to get an access token. That access token can then be used to make API calls. 1. Open the `einstein_platform.pem` file and read in the key contents. 2. Create the JWT payload. The payload is JSON that contains: - `sub`—Your Einstein Vision and Language username. You can find your username in the welcome email you receive after you get an account. If you signed up using Salesforce, your username is the email address associated with the org you signed up with. - `aud`—The API endpoint URL for generating a token. - `exp`—The expiration time in Unix time. This value is the current Unix time in seconds plus the number of seconds you want the token to be valid. For testing purposes, you can get the Unix time at [Time.is](https://time.is/Unix_time_now). The JWT payload looks like this JSON. ```json { "sub": "", "aud": "https://api.einstein.ai/v2/oauth2/token", "exp": } ``` 3. Sign the JWT payload with your RSA private key to generate an assertion. The private key is contained in the `einstein_platform.pem` file you downloaded when you signed up for an account. The code to generate the assertion varies depending on your programming language. If you're doing manual testing, you can generate an assertion using [jwt.io](https://jwt.io/). 4. Call the API and pass in the assertion. You pass in all the necessary data in the `-d` parameter. Replace `` with the assertion you generated. This cURL command shows the call. ```cURL curl -H "Content-type: application/x-www-form-urlencoded" -X POST https://api.einstein.ai/v2/oauth2/token -d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=" ``` The response looks similar to this JSON. ```json { "access_token": "EPMSDXBQSG6YH23HUE6VTA2UC53MBOEFTUND7QWVQIFOAZU42BGIK3SIXKGEWYPKO3GUFLCLHX5ZBMPPQIB6DCP2NAT7HQ108TLRQ7A", "token_type": "Bearer", "expires_in": "120" } ``` 5. Use the access token to make an API call. For example, this cURL command to get a dataset. ```cURL curl -X GET -H "Authorization: Bearer EPMSDXBQSG6YH23HUE6VTA2UC53MBOEFTUND7QWVQIFOAZU42BGIK3SIXKGEWYPKO3GUFLCLHX5ZBMPPQIB6DCP2NAT7HQ108TLRQ7A" -H "Cache-Control: no-cache" https://api.einstein.ai/v2/vision/datasets/1008108 ``` Use the access token to make any authenticated API calls as long as the token is valid (it's not expired). When the access token expires, you repeat this process to generate a new one. # Get DATETIME Values in the Right Time Zone When you call the Einstein NER (beta) endpoint and pass the `timezone` parameter, that parameter value takes precedence, and any DATETIME entities returned by the API are converted into that time zone. If you omit the `timezone` parameter, the DATETIME entity is normalized based on the `language` parameter value. This table contains the time zones associated with each language. | Language | Language code | Time zone | | --------------------- | ------------- | ------------------- | | Brazilian Portuguese | `pt_BR` | America/Sao Paulo | | Chinese (Simplified) | `zh_CN` | Asia/Shanghai | | Chinese (Traditional) | `zh_TW` | Asia/Taipei | | Danish | `da` | Europe/Copenhagen | | Dutch | `nl_NL` | Europe/Amsterdam | | English UK | `en_GB` | Europe/London | | English US | `en_US` | America/Los Angeles | | French | `fr` | Europe/Paris | | German | `de` | Europe/Berlin | | Italian | `it` | Europe/Rome | | Japanese | `ja` | Asia/Tokyo | | Korean | `ko` | Asia/Tokyo | | Portuguese | `pt_PT` | Europe/Lisbon | | Russian | `ru` | Europe/Moscow | | Spanish | `es` | Europe/Madrid | | Swedish | `sv` | Europe/Stockholm | If you specify a language but want to return the DATETIME entities in a different time zone, then you must pass the `timezone` parameter. For example, if you specify a language of English US, but you want to return DATETIME entities in the Eastern time zone, pass the `timezone` parameter with a value of America/New_York. To specify the time zone, see this list of [tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for valid time zones, and use the TZ database name value from the table. For example, `Africa/Abidjan`. ## Examples ### Specify Only a Time Zone Let’s say the current time is 1:00 pm (13:00) in the America/Los Angeles time zone. The following cURL call and the JSON response show the response if you pass a `timezone` parameter of Europe/London. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "timezone=Europe/London" -F "document=The meeting starts in two hours." https://api.einstein.ai/v2/language/entities ``` ```json { "probabilities": [ { "probability": 0.99119055, "label": "DATETIME", "token": "in two hours", "span": { "start": 19, "end": 31 }, "normalizedData": [ { "value": "2021-04-14T23:00:00.000+01:00", "unit": "minute", "type": "value" } ] } ], "object": "predictresponse" } ``` ### Specify Only a Language If you omit the `timezone` parameter, the DATETIME entity is normalized based on the `language` parameter value. Here's what the call and response looks like if you pass a language value of `en_GB`. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "language=en_GB" -F "document=The meeting starts in two hours." https://api.einstein.ai/v2/language/entities ``` ```json { "probabilities": [ { "probability": 0.99119055, "label": "DATETIME", "token": "in two hours", "span": { "start": 19, "end": 31 }, "normalizedData": [ { "value": "2021-04-14T23:00:00.000+01:00", "unit": "minute", "type": "value" } ] } ], "object": "predictresponse" } ``` Here's what the call and response looks like if you pass a value of `en_US`. The time zone associated with `en_US` is America/Los Angeles. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "language=en_US" -F "document=The meeting starts in two hours." https://api.einstein.ai/v2/language/entities ``` ```json { "probabilities": [ { "probability": 0.99119055, "label": "DATETIME", "token": "in two hours", "span": { "start": 19, "end": 31 }, "normalizedData": [ { "value": "2021-04-14T15:00:00.000-07:00", "unit": "minute", "type": "value" } ] } ], "object": "predictresponse" } ``` ### Omit Language and Time Zone If you omit the `language` parameter, the default language is `en_US`, and the DATETIME value is converted to the America/Los Angeles time zone. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "document=The meeting starts in two hours." https://api.einstein.ai/v2/language/entities ``` ```json { "probabilities": [ { "probability": 0.99119055, "label": "DATETIME", "token": "in two hours", "span": { "start": 19, "end": 31 }, "normalizedData": [ { "value": "2021-04-14T15:00:00.000-07:00", "unit": "minute", "type": "value" } ] } ], "object": "predictresponse" } ``` # How to Handle Out-of-Domain Text Einstein Intent lets you create a model that handles predictions for unexpected text. The difference between the `intent` and `multilingual-intent` algorithms as opposed to the `multilingual-intent-ood algorithm` is how a model created with each algorithm handles predictions for text that doesn’t fall into one of the labels in the model. For example, you have a case routing dataset with five labels. - Billing - Order Change - Password Help - Sales Opportunity - Shipping Info You create a model using the `intent` or `multilingual-intent` algorithm, and you send the text “what is the weather in los angeles?” for prediction. Even though the intent of this text doesn’t match any of the intent labels in the model, the prediction response returns a high probability for one of the labels in the dataset. In this case, the model returns a high probability for Sales Opportunity. ```json { "probabilities": [ { "label": "Sales Opportunity", "probability": 0.9987062 }, { "label": "Shipping Info", "probability": 0.0008089547 }, { "label": "Order Change", "probability": 0.00046194126 }, { "label": "Billing", "probability": 0.000021637188 }, { "label": "Password Help", "probability": 0.0000012197639 } ], "object": "predictresponse" } ``` It would be great if the model could identify that the text doesn't fall into any of the labels. Now you can create a model that does. Say that you use the same data to create a model but now you use the `multilingual-intent-ood` algorithm when you train the dataset and create the model. Then you send the text “what is the weather in los angeles?” for prediction. The intent of the text doesn’t match any of the existing labels, so the prediction response returns an empty array, and looks like this JSON. ```json { "probabilities": [], "object": "predictresponse" } ``` # How to Implement Multiple Languages You follow the same process to create a dataset and model that supports a specific language, but you pass in different parameters. There are two new API parameters that enable multilanguage support: - `language` - `algorithm` You use the `language` parameter when you create a dataset. The text in the dataset source file (.csv, .tsv, or .json) should be in one of the supported languages. Then, when you make the API call to create the dataset, pass in the `language` parameter that contains the value of one of the supported languages. If you don’t pass the parameter, the dataset language defaults to English (en_US). **note:** Multilanguage support is currently available only for Einstein Intent datasets and models. Multilanguage support is currently available only for Einstein Intent datasets and models." } After you create the dataset, you use the `algorithm` parameter when you train a dataset to create a model. This optional parameter specifies the algorithm used to train the dataset. Valid values are: - `intent`—Uses the original training algorithm. The training process defaults to this algorithm if the `algorithm` parameter isn’t supplied. - `multilingual-intent`—Uses the training algorithm that supports multiple languages. A model created using this algorithm returns one of the model labels, even if the text sent for prediction doesn’t fall into any of the labels. - `multilingual-intent-ood`—Uses the training algorithm that supports multiple languages and handles out-of-domain text. A model created using this algorithm returns an empty `probabilities` array when text sent for prediction doesn’t fall into one of the labels. **tip:** Even if your dataset language is English, we recommend that you use either the `multilingual-intent` or `multilingual-intent-ood` algorithm for better results. # Introduction to Salesforce Einstein Language **important:** Salesforce is retiring Einstein Vision and Language, and your subscription will not be renewable after May 1, 2024. You can continue using Einstein Vision and Language through your Order Term. To avoid interruptions, transition any Salesforce org using Einstein Vision and Language to another solution before your subscriptions expire. We will be communicating additional details to you soon. If you have questions, contact your Account Executive about your subscriptions. Create natural language processing models to classify the intent of text or to classify text as positive, negative, and neutral. Use the Einstein Language APIs to build natural language processing into your apps. Einstein Language includes three APIs that you can use to unlock powerful insights within text. - Einstein Sentiment - Classify the sentiment of text into positive, negative, and neutral classes to understand the feeling behind text. You can use the Einstein Sentiment API to analyze emails, social media, and text from chat to: - Identify the sentiment of a prospect’s emails to trend a lead or opportunity up or down. - Provide proactive service by helping dissatisfied customers first or extending promotional offers to satisfied customers. - Use trending sentiment to identify product deficiencies and measure overall satisfaction or dissatisfaction with your products. - Monitor the perception of your brand across social media channels, identify brand evangelists, and monitor customer satisfaction. Currently, Einstein Sentiment supports only English. You can create your own custom model or use our pre-built sentiment model. See [Use the Pre-Built Sentiment Model](https://developer.salesforce.com/docs/analytics/einstein-vision-language/Use-Prebuilt-Language-Models/use-pre-built-models-sentiment.md). - Einstein Intent - Categorize unstructured text into user-defined labels to better understand what users are trying to accomplish. Leverage the Einstein Intent API to analyze text from chats to: - Determine what products prospects are interested in and send customer inquiries to the appropriate sales person. - Route service cases to the correct agents or departments, or provide self-service options. Einstein Intent supports multiple languages. See [How to Implement Multiple Languages](https://developer.salesforce.com/docs/analytics/einstein-vision-language/Multilanguage-Support/how-to-implement-multiple-languages.md). ![IntentSentimentFlow](https://developer.salesforce.com/media/intent-and-sentiment-flow.png) - Einstein NER (Named Entity Recognition) - Identify entities in unstructured text. You send text to the NER model via the REST API endpoint and receive a JSON response. The NER model supports these entities: date/time, duration, email, location, money, number, organization, percent, person, phone number, and URL. ## Einstein Vision and Language Are HIPAA Compliant Einstein Vision and Language are Health Insurance Portability and Accountability Act (HIPAA) compliant. Einstein Vision and Language meet the strict HIPAA standards for processing customers’ protected health information (PHI). HIPAA compliance positions Einstein Vision and Language services to provide capabilities that can be used in the highly regulated healthcare sector. # Journey to Deep Learning ![EinsteinAsTrailblazer](https://developer.salesforce.com/media/EinsteinAsATrailblazerRightSmall.png) | PLAN | ROLL OUT | OPTIMIZE | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------- | | Get started by understanding data science and how the deep-learning development life cycle differs from the software development life cycle.
  • [Data Science Basics](https://developer.salesforce.com/docs/analytics/einstein-vision-language/Additional-Resources/data-science-basics.md)
  • [Deep Learning Development Cycle](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/deep-learning-dev-cycle.md)
  • [Select the Best Use Case](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/select-the-best-use-case.md)
| # Manage Development Einstein Vision and Einstein Language include REST-based APIs to help you quickly build natural language processing into your apps. These APIs have a few key differences from other Salesforce APIs, like the REST API or the Chatter REST API. - The API endpoints are outside of your Salesforce app. For example, the Salesforce REST API endpoint to get an account might look like: `https://yourInstance.salesforce.com/services/data/v44.0/sobjects/Account/` The Einstein Vision API endpoint to get a dataset might look like: `https://api.einstein.ai/v2/language/datasets/` - Einstein Language uses OAuth 2.0 JWT bearer token flow for authorization. This authorization method is different from the authorization methods you can use with the Salesforce REST APIs, which are OAuth 2.0 web server flow, user-agent flow, or username-password flow. Each Einstein API call requires a valid access token. You programmatically handle getting an access token by using your key or by generating a refresh token with your key to get an access token. When you use the Einstein Vision or Language APIs, calls to get a token are made to the endpoint: `https://api.einstein.ai/v2/oauth2/token` - When you customize the Salesforce app, you can use a sandbox environment, which is typically a copy of your production org, to test your changes. Because the Einstein Vision and Language APIs are outside of Salesforce, you define your own testing and deployment environments. For example, you could have some test models that you create using an account for testing. Then when you decide one of those models is ready for production, you can create that dataset and model using your production account. # Model Confusion Matrix Building a model is just the beginning of the deep-learning journey. After you create a deep-learning model, you want to know how effective it is—how well it solves the problem that you identified when you began. Evaluating the effectiveness of a model involves analyzing the model metrics. The terminology and concepts behind model metrics can be confusing at first, so we’ll try to demystify these concepts by using your actual data and real-world examples. Then you can apply these techniques to better understand your own Einstein Vision or Language model. - [Model Confusion Matrix](#model-confusion-matrix) - [Confusion Matrix](#confusion-matrix) - [Build the Confusion Matrix](#build-the-confusion-matrix) - [Get the Data](#get-the-data) - [Build the Matrix](#build-the-matrix) - [How to Read the Matrix](#how-to-read-the-matrix) - [What the Confusion Matrix Tells You](#what-the-confusion-matrix-tells-you) - [Manually Scan Your Data](#manually-scan-your-data) - [Identify Misclassified Images](#identify-misclassified-images) - [Testing Versus Training Data](#testing-versus-training-data) - [A More Advanced Example](#a-more-advanced-example) - [Build the Confusion Matrix for an Out-of-Domain Model](#build-the-confusion-matrix-for-an-out-of-domain-model) ## Confusion Matrix The confusion matrix shows you how well each label (also called a class) in your model performs. You can use this information to evaluate whether you need to add more data to a label or add a different type of data to a label. When you train a model, the training process randomly selects and holds out some of the data. After the model is created, the training process validates the model by sending the holdout data for prediction. This validation process creates the confusion matrix values. It can be difficult to interpret the confusion matrix and understand what to do with the information that it provides These steps walk you through the entire process and assume no prior knowledge. ## Build the Confusion Matrix The first step is to get the model metrics data for your model and build the confusion matrix from that data. The model metrics and confusion matrix data returned by the API vary based on the model type. This process covers the confusion matrix for models with a `modelType` of: - text-intent - text-sentiment ### Get the Data To build the confusion matrix for your model, you make an API call to get the metrics. 1. Identify the correct model metrics endpoint. **Einstein Language** GET `https://api.einstein.ai/v2/language/models/` The response from this call differs based on the type of dataset that the model was created from: `text-intent` or `text-sentiment`. However, the model metrics for a language model always contains a `labels` array and a `confusionMatrix` array. 2. Replace `` with the ID of your model, and use cURL or Postman to make the API call. The cURL call to get the model metrics looks like this. ```cURL curl -X GET -H "Authorization: Bearer " https://api.einstein.ai/v2/language/models/ZWUH3K5NVLQNWPWWSKDLBCXQXU ``` The call to get the model metrics returns a response similar to this JSON. ```json { "createdAt": "2019-06-05T18:09:37.000+0000", "object": "metrics", "metricsData": { "f1": [0.8333333333333333, 0.8333333333333333], "labels": ["Mountains", "Beaches"], "testAccuracy": 0.8333, "trainingLoss": 0.0521, "confusionMatrix": [ [5, 0], [2, 5] ], "trainingAccuracy": 0.981 }, "id": "ZWUH3K5NVLQNWPWWSKDLBCXQXU" } ``` ### Build the Matrix Now that you have the model metrics data for your model, you use that data to build the confusion matrix. As you step through this process, you build the matrix manually. But you use the same logic to create the array programmatically. 1. When you look at the `labels` array in the response JSON, you see two labels in this model. Add one to the number of labels and then create a table with that number of columns and rows. In this case, the table has three rows and three columns. | | | | | --- | --- | --- | | | | | | | | | 2. Enter each value in the labels array in the column and row headers in the same order that the labels appear in the array. Your matrix now looks like this table. | | Mountains | Beaches | | ------------- | --------- | ------- | | **Mountains** | | | | **Beaches** | | | 3. Look at the `confusionMatrix` array in the JSON response. It contains an array of numbers for each label. The arrays match the order of the labels: the first array contains predictions for the examples labeled Mountains, and the second array has predictions for the examples labeled Beaches. Add these numbers to the matrix. | | Mountains | Beaches | | ------------- | --------- | ------- | | **Mountains** | 5 | 2 | | **Beaches** | 0 | 5 | Now your confusion matrix is complete! ## How to Read the Matrix To make the confusion matrix easier to read, one trick is to add a row to the top with the heading **Actual Label** and a column to the left with heading **Label the Model Predicted**. After you get more familiar with the confusion matrix, you might not need to add the extra labels. 1. Across the top, are the actual labels (Mountains, Beaches) of the validation examples from your dataset. 2. On the left side, are the labels (Mountains, Beaches) that the model predicted for those examples. ![ConfusionMatrixHeadings](https://developer.salesforce.com/media/conf_matrix_headings.png) Of the 12 examples that were held out for validation and have the actual label Mountains, the model predicted that: - 5 are in the label Mountains - 0 are in the label Beaches Of the six examples that were held out for validation and have the actual label Beaches, the model predicted that: - 5 are in the label Beaches - 2 are in the label Mountains The bold numbers in the matrix indicate the correct predictions. The correct predictions are always positioned in the diagonal cells of the table, which makes it easier to read. | | Mountains | Beaches | | ------------- | --------- | ------- | | **Mountains** | **5** | 2 | | **Beaches** | 0 | **5** | ## What the Confusion Matrix Tells You The confusion matrix gives you a visual explanation of your model’s accuracy. For the test data, it shows you how many examples it predicted correctly and incorrectly and which label it predicted the examples were in. This confusion matrix is based on a small dataset. But you can still see that the model was confused about two images: It predicted that two examples labeled Beaches belong in the Mountains class. The model gave incorrect predictions for two examples that are labeled Beaches. This is quite a high percentage (about 40%), given that this is a small dataset. Now what can you do with this information? Here are some guidelines on how to use the confusion matrix and possible issues that it identifies. ### Manually Scan Your Data The model is created from your labeled data, so the first thing to do is to go back to your data. With a small dataset, an easy step to take is to manually scan the data in the labels that the model is confusing. Check whether any patterns emerge from the source data or how it’s labeled. Sometimes data is added with the wrong label, which you can fix by relabeling the data and creating a new dataset and model. ### Identify Misclassified Images The model metrics don’t tell you which examples were held out for validation and which of those examples were misclassified. After you create a model, you can programmatically identify the misclassified images by taking all the images from the dataset that you used to create the model and sending them in for a prediction. You call one of these endpoints depending on the model type. - image: `/predict` - text-intent: `/intent` - text-sentiment: `/sentiment` After you identify the misclassified images, you can analyze them to see what the issue might be. Were the images mislabeled before they were added to the dataset? In the beaches and mountains scenario, do the misclassified images contain both a beach and a mountain? If so, you could consider removing the misclassified images from your dataset. On the other hand, if you think images that are similar to the misclassified images will come in for prediction—for example, images with both beaches and mountains—consider adding a new label and corresponding examples to your dataset and model. When the confusion matrix reports misclassification, try adding more data to the misclassified labels. The beach-mountain dataset has only two labels, so you could add more data to both labels and then re-evaluate the confusion matrix. ## Testing Versus Training Data When you train a dataset to create a model, the training process holds out a percentage of the data for validating the model and uses the rest to create the model. The ratio of test data versus training data is called the split ratio. When you call the `/train` endpoint, the training process test data percentage defaults to 10 percent for vision models and 20 percent for language models. You can also specify this ratio by using the `trainSplitRatio` parameter. To get more information about test data and training data, use the API call to get the training status. The call looks like this cURL command. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/vision/train/ZWUH3K5NVLQNWPWWSKDLBCXQXU ``` From the following response, you can see: - 99 examples (images) in total - 87 examples used to train the dataset and create the model - 12 examples held out to validate the model ```json { "datasetId": 1017867, "datasetVersionId": 13829, "name": "Confusion Matrix Test Model", "status": "SUCCEEDED", "progress": 1, "createdAt": "2019-06-05T18:08:27.000+0000", "updatedAt": "2019-06-05T18:10:07.000+0000", "learningRate": 0.001, "epochs": 3, "object": "training", "modelId": "ZWUH3K5NVLQNWPWWSKDLBCXQXU", "trainParams": null, "trainStats": { "labels": 2, "examples": 99, "averageF1": 0.7482517482517483, "totalTime": "00:01:37:976", "transforms": { "toFlip": true, "toRotate": false, "cropLocation": "random" }, "trainingTime": "00:01:35:986", "earlyStopping": false, "lastEpochDone": 3, "modelSaveTime": "00:00:01:049", "testSplitSize": 12, "trainSplitSize": 87, "datasetLoadTime": "00:00:01:989", "preProcessStats": null, "postProcessStats": null }, "modelType": "image" } ``` Going back the confusion matrix, if you total up all the numbers in the matrix, that number is equal to the `testSplitSize`—the number of examples held out to test the model. ## A More Advanced Example Now let’s look at an example from a model with more labels. You can create this Einstein Intent model by following the steps in [Einstein Intent Quick Start](https://developer.salesforce.com/docs/analytics/einstein-vision-language/Einstein-Language/intent-quick-start-custom-classifier.md). The cURL call to get the model metrics looks like this. ```cURL curl -X GET -H "Authorization: Bearer " https://api.einstein.ai/v2/language/models/626SYFRFKHMH2FEX3ND77FXG2I ``` The call to get the model metrics returns a response similar to this JSON. The precision recall curve data has been removed for brevity. ```json { "createdAt": "2019-06-11T19:17:25.000+0000", "object": "metrics", "metricsData": { "f1": [0.6153846153846154, 0.875, 1, 0.6666666666666666, 1], "labels": ["Order Change", "Sales Opportunity", "Billing", "Shipping Info", "Password Help"], "recall": [], "macroF1": 0.7222222222222223, "testLoss": 0.44152334332466125, "precision": [], "threshold": { "Billing": 0.83269, "Order Change": 0.89199, "Password Help": 0.75623, "Shipping Info": 0.82936, "Sales Opportunity": 0.90116 }, "testAccuracy": 0.774193525314331, "trainingLoss": 0.020516589283943176, "confusionMatrix": [ [4, 0, 0, 1, 0], [0, 7, 0, 1, 0], [0, 0, 2, 0, 0], [4, 1, 0, 7, 0], [0, 0, 0, 0, 4] ], "trainingAccuracy": 1, "precisionRecallCurve": {} }, "id": "626SYFRFKHMH2FEX3ND77FXG2I" } ``` Follow the previous steps to create the confusion matrix. It looks like this table. | | Order Change | Sales Opportunity | Billing | Shipping | Password Help | | --------------------- | ------------ | ----------------- | ------- | -------- | ------------- | | **Order Change** | **4** | 0 | 0 | 4 | 0 | | **Sales Opportunity** | 0 | **7** | 0 | 1 | 0 | | **Billing** | 0 | 0 | **2** | 0 | 0 | | **Shipping Info** | 1 | 1 | 0 | **7** | 0 | | **Password Help** | 0 | 0 | 0 | 0 | **4** | This confusion matrix points to a few areas in the data that you might want to investigate. First, if you look at the Shipping Info column, you see that the model is misclassifying a large percentage of text labeled “Shipping Info.” In particular, the model is classifying quite of few of those examples (4 out of 12) as “Order Change.” For this case, it would be helpful to send all the examples used to create the dataset into the model for prediction to identify which examples were misclassified. Then you can use that information to identify patterns in the data or labeling that requires correction. ## Build the Confusion Matrix for an Out-of-Domain Model When you train an intent dataset and use the `multilingual-intent-ood` algorithm, the model handles text that doesn't fit into any one of the model labels. Text that doesn't fit into any of the model labels is also called out-of-domain (OOD) text. There are some differences in the model metrics and learning curve data for an OOD model. - The f1 score and all other metrics are returned only for the model labels. - The confusion matrix includes numbers for the labels in the model as well as for the OOD label. However, the `labels` array includes only labels in the model. It doesn't include an OOD label. Building the confusion matrix for an OOD model requires one extra step. When you build the confusion matrix for an OOD model, you must manually add a label to the `labels` array. You can name the label any value. Then when you loop through the numbers in the `confusionMatrix`, the last number in each array, and the last array itself, is associated with the OOD label. For example, the `labels` array for a case routing OOD model might look like this JSON. ```json "labels": [ "Shipping Info", "Sales Opportunity", "Order Change", "Password Help", "Billing" ] ``` The confusionMatrix array returns six arrays and each array has six elements. `"confusionMatrix":[[4,1,0,0,1,2],[0,7,0,0,0,0],[3,0,2,0,1,2],[0,0,0,2,1,1],[1,0,0,0,1,1],[0,0,0,0,0,0]]` When you build the confusion matrix, be sure to add a sixth label to the labels array so that the matrix is correct: Shipping Info, Sales Opportunity, Order Change, Password Help, Billing, Out of Domain. The final confusion matrix using this data looks like this table. | | Shipping Info | Sales Opportunity | Order Change | Password Help | Billing | OOD | | --------------------- | ------------- | ----------------- | ------------ | ------------- | ------- | ----- | | **Shipping Info** | **4** | 1 | 0 | 0 | 1 | 1 | | **Sales Opportunity** | 0 | **7** | 0 | 0 | 0 | 0 | | **Order Change** | 3 | 0 | **2** | 0 | 1 | 1 | | **Password Help** | 0 | 0 | 0 | **2** | 1 | 1 | | **Billing** | 1 | 0 | 0 | 0 | **1** | **1** | | **OOD** | 0 | 0 | 0 | 0 | 0 | 0 | # Passport This table lists the entities returned by the OCR model for a passport. Supports passports only from Australia, Canada, United Kingdom, and United States. | Entity name | Additional info | Possible form keys | | ----------------- | --------------------------------------------------------------------------- | -------------------------------------- | | `authority` | | Authority | | `code` | Country code. | Code | | `country` | Country code. | | | `date_of_birth` | | Date of birth | | `expiration_date` | | Date of expiration
Date of expiry | | `first_name` | Returned if the first name is on a separate line. | Given name. | | `gender` | | Sex | | `issue_country` | In some passports, the issuing country is in the Code field. | Issuing Country | | `issue_date` | | Date of issue
Date of Issue | | `last_name` | Returned if the first name is on a separate line. | Surname | | `name` | Returns the full name. | Name | | `nationality` | | Nationality | | `number` | | Passport No.
DOCUMENT No. | | `place_of_birth` | Depending on the passport, this value can be a city or a state and country. | Place of birth | | `type` | Each country has their own passport types. | Type/Type | # Pay Stub This table lists the entities returned by the OCR model for a pay stub. The OCR model supports pay stubs with varying layouts. | Entity name | Possible form keys | | ------------------------ | ------------------------------------------ | | `account_name` | | | `account_number` | | | `allowance` | Exemptions
Allowance | | `bank_name` | | | `check_number` | Check No. | | `clock_number` | | | `company_address` | | | `company_name` | | | `company_phone_number` | | | `current_deductions` | Current Deductions | | `current_total` | Current Total
Gross Pay | | `department_number` | | | `employee_address` | | | `employee_id` | Employee ID | | `employee_name` | | | `federal_allowance` | | | `file_number` | | | `local_allowance` | | | `marital_status` | | | `net_pay` | Net Pay | | `pay_amount` | | | `pay_date` | Pay Date | | `pay_record` | | | `pay_to_the_order_of` | | | `payroll_number` | | | `period_beginning` | | | `period_ending` | Period Ending | | `social_security_number` | Social Security Number
Social Sec. ID | | `state_allowance` | | | `ytd_deductions` | YTD Deductions | | `ytd_gross` | YTD Gross | | `ytd_net_pay` | YTD Net Pay | # Permanent Resident Card This table lists the entities returned by the OCR model for a permanent resident card. Supports cards only from the United States, including Washington, DC. | Entity name | Form key | | ----------------- | ---------------- | | `date_of_birth` | Date of Birth | | `expiration_date` | Card Expires | | `first_name` | Given Name | | `gender` | Sex | | `last_name` | Surname | | `number` | USCIS# | | `place_of_birth` | Country of Birth | | `resident_since` | Resident Since | | `class` | Category | # Rate Limits You can call the Einstein Vision and Language APIs as often as you need to create datasets, add examples, and train models. However, there are limits to the number of prediction calls you can make. Einstein Vision and Language provides two tiers of usage: free and paid. Each tier gives you a specific number of prediction calls. These limits apply only to predictions. A prediction is any POST call to these endpoints: - `/language/intent` - `/language/entities` - `/language/sentiment` A prediction call includes both predictions made from custom classifiers and predictions made from the pre-built classifiers. - **Salesforce**—If you signed up for an account using Salesforce, contact your account executive to purchase predictions for Einstein Vision and Language. To add predictions to an existing account, you must provide the organization ID for your Einstein Vision and Language account. To get your Einstein Vision and Language organization ID, call the usage API. See [Get API Usage](/docs/analytics/einstein-vision-language/references). If you’re not sure what method you used to sign up or if you have any questions, post on the Einstein Platform [developer forum](https://developer.salesforce.com/forums?communityId=09aF00000004HMGIA2#!/feedtype=RECENT&dc=Predictive_Services&criteria=ALLQUESTIONS). ## Monitor Usage There are two ways you can get your prediction usage information from the Einstein Vision and Language APIs. - Response headers—headers that contain basic rate limit information are returned by a call to any of the prediction resources. These headers give you a way to monitor your prediction usage as prediction calls are made. - The `/apiusage` resource—contains detailed information about your limits and prediction usage. ## Rate Limit Headers Any time you make an API call to the prediction resources, rate limit information is returned in the header. The rate limit headers specify your prediction usage for the current calendar month only. ## Prediction Usage API If you want to see detailed plan information, make an explicit call to the `/apiusage` resource. See [Get API Usage](/docs/analytics/einstein-vision-language/references). # Release Notes Find out what's new, changed, or deprecated in the Einstein Vision and Language APIs. **note:** The Einstein Vision and Language release notes are now in one place. For the latest release notes, see the [Salesforce Release Notes](https://help.salesforce.com/articleView?id=release-notes.salesforce_release_notes.htm). At the top of the page, you can select the release. ## Release Notes Archive For previous years' release notes, see [Release Notes Archive](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/release-notes-archive.md). # Release Notes Archive This page contains all previous years' release notes. ## July 31, 2023 ### New The old api.metamind.io endpoint has been retired. See note from [July 27, 2017](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/release-notes-archive.html#the-api-now-uses-the--endpoint) ### KNOW ISSUE #### API Usage call returns inaccurate data **What:** There is an issue with API Usage tracking that return the wrong number of used API calls. To ensure continued access to the service are temporarily not enforcing rate limits while work to rectify the API Usage issue. Expected date for the resumption of accurate API Usage and enforcement of rate limting is first quarter 2024. ### DEPRECATED #### Use Feedback to improve a model **What:** To enhance the security and availability of Salesforce services, this endpoint is now **deprecated**, and **we plan to remove it on August 14, 2023**. This endpoint is not available to new customers, and we discourage existing customers from using it. #### Get Deletion Status **What:** To simplify Einstein Language API the **Get Deletsion Status** endpoint has been deprecated. Deletion is now immediate. To determine if the a dataset has been deleted use the [Get Dataset](/docs/analytics/einstein-vision-language/references) command. A deleted dataset will return a 404 (Not Found) http status. #### Upload Dataset from URL **What:** To enhance the security and availability of Salesforce services, the `path` request parameter on the dataset upload endpoint has been deprecated. ### New #### #### Einstein Intent now defaults to multilingual-text algorithm **What:** Specifying either `intent` or `multilingual-intent` will default to `multilingual-intent`. The `intent` algorithm was V2, which has been deprecated. The default `multilingual-intent` algorithm uses V3 with has improvements in accuracy and performnce. You will not need to make any changes to existing calls that use `intent` as the algorithm. #### Language API model metrics availability **What:** A side effect of the migration of Einstein Vision and Language services to Hyperforce, calls to **Get Model Metrics** (`https://api.einstein.ai/v2/language/models/` will return a 404 (Not found) error until the next dataset train or retrain. ## May 6, 2020 ### NEW #### Detect products on retail shelves with an optimized algorithm. **How:** To use the retail execution algorithm, first create a dataset that has a type of `image-detection`. Then when you train the dataset to create a model, you specify an algorithm of `retail-execution`. The cURL command is as follows. ```cURL curl -X POST -H "Authorization: " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Alpine Retail Model" -F "datasetId=" -F "algorithm=retail-execution" https://api.einstein.ai/v2/language/train ``` These Einstein Vision calls take the `algorithm` parameter. - Train a model—POST `/v2/vision/train` - Retrain a model—POST `/v2/vision/retrain` ## May 4, 2020 ### NEW #### Detect text in an image with Einstein OCR (Generally Available) Get optical character recognition (OCR) models that detect alphanumeric text in an image with Einstein OCR. You access the models from a single REST API endpoint. Each model has specific use cases, such as business card scanning, product lookup, and digitizing documents and tables. **How:** When you call the API, you send in an image, and the JSON response contains various elements based on the value of the task parameter. Here’s what a cURL call to the OCR endpoint looks like. ```cURL curl -X POST -H "Authorization: Bearer " -F sampleLocation="https://www.publicdomainpictures.net/pictures/240000/velka/emergency-evacuation-route-signpost.jpg" -F task="text" -F modelId="OCRModel" https://api.einstein.ai/v2/vision/ocr ``` The response JSON returns the text and coordinates of a bounding box (in pixels) for that text. ```json { "task": "text", "probabilities": [ { "probability": 0.99937266, "label": "ROUTE", "boundingBox": { "minX": 582, "minY": 685, "maxX": 1151, "maxY": 815 } }, { "probability": 0.99471515, "label": "EMERGENCY", "boundingBox": { "minX": 361, "minY": 208, "maxX": 1383, "maxY": 346 } }, { "probability": 0.99469215, "label": "EVACUATION", "boundingBox": { "minX": 331, "minY": 438, "maxX": 1401, "maxY": 570 } } ], "object": "predictresponse" } ``` #### Einstein Intent now supports multiple languages. Einstein Intent datasets and models now support these languages: English (US), English (UK), French, German, Italian, Portuguese, Spanish, Chinese (Simplified) (beta), Chinese (Traditional) (beta), Japanese (beta). You specify the language when you create an intent dataset. When you train that dataset, the model inherits the language of the dataset. **How:** There are two new API parameters that enable multilanguage support: `language` and `algorithm`. When you create the dataset, you specify the language in the `language` parameter. When you train the dataset to create a model, you pass in the `algorithm` parameter with a value of `multilingual-intent` or `multilingual-intent-ood` (to create a model that handles out-of-domain predictions). These calls take the `language` parameter. - Create a dataset asynchronously—POST `/v2/language/datasets/upload` - Create a dataset synchronously—POST `/v2/language/datasets/upload/sync` These calls take the `algorithm` parameter. - Train a dataset—POST `/v2/language/train` - Retrain a dataset—POST `/v2/language/retrain` **note:** As a beta feature, Chinese (Simplified), Chinese (Traditional), and Japanese language support is a preview and isn’t part of the “Services” under your master subscription agreement with Salesforce. Use this feature at your sole discretion, and make your purchase decisions only on the basis of generally available products and features. Salesforce doesn’t guarantee general availability of this feature within any particular time frame or at all, and we can discontinue it at any time. This feature is for evaluation purposes only, not for production use. It’s offered as is and isn’t supported, and Salesforce has no liability for any harm or damage arising out of or in connection with it. All restrictions, Salesforce reservation of rights, obligations concerning the Services, and terms for related Non-Salesforce Applications and Content apply equally to your use of this feature. #### Create Einstein Intent models that support out-of-domain text. Einstein Intent lets you create a model that handles predictions for unexpected, out-of-domain, text. Out-of-domain text is text that doesn’t fall into any of the labels in the model. **How:** When you train an intent dataset, pass the `algorithm` parameter with a value of `multilingual-intent-ood`. To see how the algorithm works, let’s say you have a case routing model with five labels: Billing, Order Change, Password Help, Sales Opportunity, and Shipping Info. The following text comes in for prediction: “What is the weather in Los Angeles?” If the model was created using the standard algorithm, the response looks like this JSON. ```json { "probabilities": [ { "label": "Sales Opportunity", "probability": 0.9987062 }, { "label": "Shipping Info", "probability": 0.0008089547 }, { "label": "Order Change", "probability": 0.00046194126 }, { "label": "Billing", "probability": 0.000021637188 }, { "label": "Password Help", "probability": 0.0000012197639 } ], "object": "predictresponse" } ``` The text sent for prediction clearly doesn’t fall into any of the labels. The model isn’t designed to handle predictions that don’t match one of the labels, so the model returns the labels with the best probability. If you create the model with the `multilingual-intent-ood` algorithm, and you send the same text for prediction, the response returns an empty `probabilities` array. ```json { "probabilities": [], "object": "predictresponse" } ``` These calls take the `algorithm` parameter. - Train a dataset—POST `/v2/language/train` - Retrain a dataset—POST `/v2/language/retrain` ## December 12, 2019 ### NEW #### Get more detailed error messages for Einstein Objection Detection training API calls. When you train an object detection dataset and the training process encounters an error, the API now returns more descriptive error messages. In most cases, the error message specifies the issue that caused the error and how to fix it. **How**: The improved errors are returned for these API endpoints when the dataset type is `image-detection`. - Train a dataset—POST `/v2/vision/train` - Retrain a dataset—POST `/v2/vision/retrain` ## November 21, 2019 ### NEW #### Added elements in Language API model metrics response. New elements returned in the model metrics let you better understand the performance of your model. The response JSON for an Einstein Language API call that returns model metrics information contains three new elements: the `macroF1` field, the `precision` array, and the `recall` array. **When:** This change applies to all language models created after September 30, 2019. If you want to see these changes for models created before that date, retrain the dataset and create a new model. **How:** The new field and arrays appear in the response for these calls when the model type is `text-intent` or `text-sentiment`. - Get model metrics—GET `/v2/language/models/` - Get model learning curve—GET `/v2/language/models//lc` ## October 14, 2019 ### CHANGED #### Text datasets can contain up to 3 million records. The maximum number of words in a text dataset is now 3 million. A text dataset is a dataset that has a type of `text-intent` or `text-sentiment`. **How:** You receive an error from the following calls when you train a text dataset that has more than 3 million words: - Train a dataset—POST `/v2/language/datasets/train` - Retrain a dataset—POST `/v2/language/datasets/retrain` To avoid this error, be sure that when you create a dataset or add examples to a dataset, that it contains less than 3 million words across all examples. For best results, we recommend that each example is around 100 words. ## October 4, 2019 ### CHANGED #### Object detection max image size increased. We increased the maximum size of an image you can add to an object detection dataset from 1 MB to 5 MB. **How:** The new maximum image size applies to these calls when the dataset type is image-detection. - Create a dataset asynchronously—POST `/v2/vision/datasets/upload` - Create a dataset synchronously—POST `/v2/vision/datasets/upload/sync` - Create examples from a .zip file—PUT `/v2/vision/datasets//upload` - Create an example—POST `/v2/vision/datasets//examples` - Create a feedback example—POST `/v2/vision/feedback` - Create feedback examples from a .zip file—PUT `/v2/vision/bulkfeedback` ## July 15, 2019 ### NEW #### Intent API response JSON contains a new `algorithm` field. The response JSON for an Einstein Intent API call that returns model information now contains the `algorithm` field. The default return value is `intent`. **How:** The `algorithm` field appears in the response for these calls when the dataset type or model type is `text-intent`. - Train a dataset—POST `/v2/language/datasets/train` - Retrain a dataset—POST `/v2/language/datasets/retrain` - Get training status—GET `/v2/language/train/` - Get model metrics—GET `/v2/language/models/` - Get all models for a dataset—GET `/v2/language/datasets//models` ## July 3, 2019 ### NEW #### API response JSON contains a new `language` field. The response JSON for an Einstein Vision API call that returns model information now contains the `language` field. When you train a dataset, the resulting model inherits the language of the dataset. For Einstein Vision datasets and models, the return value is `N/A`. **How:** The `language` field appears in the response for these calls. - Train a dataset—POST `/v2/vision/train` - Retrain a dataset—POST `/v2/vision/retrain` - Get training status—GET `/v2/vision/train/` - Get model metrics—GET `/v2/vision/models/` - Get all models for a dataset—GET `/v2/vision/datasets//models` #### API response JSON contains a new language field. The response JSON for an Einstein Language API call that returns model information now contains the `language` field. When you train a dataset, the resulting model inherits the language of the dataset. For Einstein Vision datasets and models, the return value is `en_US`. **How:** The `language` field appears in the response for these calls. - Train a dataset—POST `/v2/language/datasets/train` - Retrain a dataset—POST `/v2/language/datasets/retrain` - Get training status—GET `/v2/language/train/` - Get model metrics—GET `/v2/language/models/` - Get all models for a dataset—GET `/v2/language/datasets//models` #### Object Detection API response JSON contains a new `algorithm` field. The response JSON for an Einstein Vision API call that returns object detection model information now contains the `algorithm` field. The default return value is `object-detection`. **How:** The `algorithm` field appears in the response for these calls when the dataset type or model type is `image-detection`. - Train a dataset—POST `/v2/vision/train` - Retrain a dataset—POST `/v2/vision/retrain` - Get training status—GET `/v2/vision/train/` - Get model metrics—GET `/v2/vision/models/` - Get all models for a dataset—GET `/v2/vision/datasets//models` #### Einstein Language default `language` is now `en-US`. The default language changed to `en_US` from `ENGLISH`. **How:** The `language` field now contains the value `en_US` in the response for these calls. - Create a dataset asynchronously—POST `/v2/language/datasets/upload` - Create a dataset synchronously—POST `/v2/language/datasets/upload/sync` - Get a dataset—GET `/v2/language/datasets/` - Get all datasets—GET `/v2/language/datasets` - Create examples from a file—PUT `/v2/language/datasets//upload` ## April 30, 2019 ### NEW #### Einstein Vision and Language Model Builder just released. An [AppExchange package](https://appexchange.salesforce.com/appxListingDetail?listingId=a0N3A00000Ed1V8UAJ) that provides a UI for the Einstein Vision and Language deep learning APIs. You can easily create datasets, build models, and make predictions all right from Salesforce. ## April 18, 2019 ### NEW #### API response JSON contains a new `numOfDuplicates` field. The response JSON for any Einstein Vision API call that returns dataset information now includes the `numOfDuplicates` field. This field indicates the number of images not added to the dataset because they’re duplicates. **Why:** When you create a dataset or add data to a dataset, duplicate images are omitted. The `numOfDuplicates` field appears in the response for these calls. - Create a dataset asynchronously—POST `/v2/vision/datasets/upload` - Create a dataset synchronously—POST `/v2/vision/datasets/upload/sync` - Create a dataset —POST `/v2/vision/datasets` - Get a dataset—GET `/v2/vision/datasets/` - Get all datasets—GET `/v2/vision/datasets` - Create examples from a .zip file—PUT `/v2/vision/datasets//upload` - Create feedback examples from a .zip file—PUT `/v2/vision/bulkfeedback` #### API response JSON contains a new `numOfDuplicates` field. The response JSON for any Einstein Language API call that returns dataset information now contains the `numOfDuplicates` field. This field indicates the number of text strings not added to the dataset because they’re duplicates. **Why:** When you create a dataset or add data to a dataset, duplicate text strings are omitted. The `language` field appears in the response for these calls. - Create a dataset asynchronously—POST `/v2/language/datasets/upload` - Create a dataset synchronously—POST `/v2/language/datasets/upload/sync` - Get a dataset—GET `/v2/language/datasets/` - Get all datasets—GET `/v2/language/datasets` - Create examples from a file—PUT `/v2/language/datasets//upload ## March 27, 2019 ### CHANGED #### Maximum image dataset size increased to 2 GB. We doubled the maximum size of an image dataset from 1 GB to 2 GB. #### Maximum text dataset size increased to 2 GB. We doubled the maximum size of a text dataset from 1 GB to 2 GB. ## March 21, 2019 ### CHANGED #### Number of API calls to return examples is limited to 30 calls per month. Each Einstein Vision and Language account is now limited to 30 calls per calendar month to Einstein Vision and Einstein Language endpoints that return examples. This limit applies across all APIs that return examples. If you exceed this limit, you receive an error message. **How:** These API endpoints return examples. - Get all Einstein Vision examples—GET `/v2/vision/datasets//examples` - Get all Einstein Vision examples for a label—GET `/v2/vision/examples?labelId=` - Get all Einstein Language examples —GET `/v2/language/datasets//examples` - Get all Einstein Language examples for a label—GET `/v2/language/examples?labelId=` ## March 8, 2019 ### NEW #### API response JSON contains a new `language` field. The response JSON for any Einstein Vision API call that returns dataset information now contains the `language` field. The return value is `N/A`. **How:** The `language` field appears in the response for these calls. - Create a dataset asynchronously—POST `/v2/vision/datasets/upload` - Create a dataset synchronously—POST `/v2/vision/datasets/upload/sync` - Create a dataset —POST `/v2/vision/datasets` - Get a dataset—GET `/v2/vision/datasets/` - Get all datasets—GET `/v2/vision/datasets` - Create examples from a .zip file—PUT `/v2/vision/datasets//upload` - Create feedback examples from a .zip file—PUT `/v2/vision/bulkfeedback` #### Use the optional `language` parameter when creating a text dataset. When creating a text dataset, you can now specify a language with the new `language` parameter. The default is `ENGLISH`. We created this parameter for future use, so you don’t need to do anything now. **How:** The `language` parameter is available in these calls. - Create a dataset asynchronously—POST `/v2/language/datasets/upload` - Create a dataset synchronously—POST `/v2/language/datasets/upload/sync` #### API response JSON contains a new `language` field. The response JSON for any Einstein Language API call that returns dataset information now contains the `language` field. The return value for existing datasets is `ENGLISH`. **How:** The `language` field appears in the response for these calls. - Create a dataset asynchronously—POST `/v2/language/datasets/upload` - Create a dataset synchronously—POST `/v2/language/datasets/upload/sync` - Get a dataset—GET `/v2/language/datasets/` - Get all datasets—GET `/v2/language/datasets` - Create examples from a file—PUT `/v2/language/datasets//upload` ## January 24, 2019 ### CHANGED #### Exceeding the maximum dataset size returns an error. When you create a dataset (using the POST call) or add data to a dataset (using the PUT call), if the resulting dataset exceeds the maximum dataset size of 1 GB, the call fails and an error is returned. This change applies to Einstein Vision and Einstein Language. See [Create a Dataset From a File Asynchronously (Language)](/docs/analytics/einstein-vision-language/references) and [Create Examples From a File (Language)](/docs/analytics/einstein-vision-language/references). #### Training requests from customers on a paid plan are prioritized. Training requests from customers on a paid plan are prioritized before training requests made by customers on a free plan. A training request is any call to the `/train` or `/retrain` resources. Use the API usage call to find out what kind of plan you have. See [Get API Usage](/docs/analytics/einstein-vision-language/references). ``` curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/apiusage ``` The following is a list of the plans. The free plan has a value of `STARTER`. - `HEROKU` - `STARTER`—2,000 predictions per calendar month. - `BRONZE`—10,000 predictions per calendar month. - `SILVER`—250,000 predictions per calendar month. - `GOLD`—One million predictions per calendar month. - `SALESFORCE` - `STARTER`—2,000 predictions per calendar month. - `SFDC_1M_EDITION`—One million predictions per calendar month. You might see a delay in training if you're on the free tier of service and there are other training requests in the queue. This change applies to Einstein Vision and Einstein Language. ## March 23, 2018 ### CHANGED #### Changes to delete dataset functionality for Einstein Vision and Einstein Language. The delete dataset API call no longer returns a 204 status code for a successful dataset deletion. Instead, the API returns a 200 status code, which specifies that a dataset deletion response was successfully received, but the deletion has yet to be completed. See [Delete a Dataset (Language)](/docs/analytics/einstein-vision-language/references). In addition to the new status code, the call returns a JSON response with a deletion ID. You can use this ID to query the status of the deletion. ```json { "id": "Z2JTFBF3A7XKIJC5QEJXMO4HSY", "organizationId": "108", "type": "DATASET", "status": "QUEUED", "progress": 0, "message": null, "object": "deletion", "deletedObjectId": "1003360" } ``` Deleting a dataset no longer deletes the associated models. You must explicitly delete models. See [Delete a Model (Language)](/docs/analytics/einstein-vision-language/references). ### NEW #### Get the deletion status with this new API endpoint. After you delete a dataset or a model, it may take some time for the data to be deleted. To confirm whether a dataset or model has been deleted, call the `/deletion` endpoint along with the deletion ID. ``` curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/vision/deletion/ ``` Valid values are: - `QUEUED`—Object deletion hasn't started. - `RUNNING`—Object deletion is in progress. - `SUCCEEDED`—Object deletion is complete. - `SUCCEEDED_WAITING_FOR_CACHE_REMOVAL`—Object was deleted, but it can take up to 30 days to delete some related files that are cached in the system. #### Delete a model with this new API endpoint. Now deleting a dataset doesn't delete the models associated with that dataset. Instead, use this new API endpoint to delete a model. This cURL call deletes a model. ``` curl -X DELETE -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" https://api.einstein.ai/v2/language/models/ ``` The response looks similar to this JSON. ``` { "id": "2GAUJLAG3L5WFQE6GYTOM4O2IM", "organizationId": "108", "type": "MODEL", "status": "QUEUED", "progress": 0, "message": null, "object": "deletion", "deletedObjectId": "P3NDGNJFA5JG5J7RW54WUZDWGI" } ``` See [Delete a Model (Vision)](/docs/analytics/einstein-vision-language/references) and [Delete a Model (Language)](/docs/analytics/einstein-vision-language/references). After you delete a model, use the `id` to check the status of the deletion. ## March 1, 2018 ### NEW #### Reset your private key. After you sign up for an account, you download or save your private key in the form of a .pem file. But sometimes things happen. If you lose your private key, you can reset it. See [Reset Your Private Key](https://developer.salesforce.com/docs/analytics/einstein-vision-language/Additional-Resources/reset-your-private-key.md). ## February 8, 2018 ### CHANGED #### Rate limiting for Einstein Language (which includes Einstein Intent and Einstein Sentiment) and Einstein Object Detection goes into effect today. The free tier of our service will offer 2,000 free predictions (increased from 1,000 free predictions) each calendar month. See [Rate Limits](https://developer.salesforce.com/docs/analytics/einstein-vision-language/Additional-Resources/rate-limits.md). When you exceed the maximum number of predictions for the current calendar month, you receive an error message when you call one of the prediction resources. To purchase predictions, contact your Salesforce or Heroku AE. A prediction is any POST call to these endpoints: - `/language/intent` - `/language/sentiment` ## January 8, 2018 ### CHANGED #### On January 15, 2018, the response returned by the `/detect` call is changing. In the new response JSON, the field `"resultType": "DetectionResult"` is removed and the field `"object": "predictresponse"` is added. The new response looks like this JSON. ```json { "probabilities": [ { "label": "Alpine - Corn Flakes", "probability": 0.97197026, "boundingBox": { "minX": 646, "minY": 865, "maxX": 885, "maxY": 1430 } }, ...{ "label": "Alpine - Bran Cereal", "probability": 0.57089806, "boundingBox": { "minX": 921, "minY": 694, "maxX": 1257, "maxY": 1304 } } ], "object": "predictresponse" } ``` See the Detection with Image File section and Detection with Image URL section of [Detection]. ### CHANGED #### Generate an access token using a refresh token. Instead of using your private key to generate an access token, you can generate a refresh token and use that to generate an access token. A refresh token is a JWT token that never expires. A refresh token is useful in cases where an application is offline and doesn't have access to they key, such as mobile apps. See [Generate an OAuth Access Token](/docs/analytics/einstein-vision-language/references). ## December 19, 2017 ### NEW #### Einstein Sentiment, Einstein Intent, and Einstein Object Detection now generally available. Einstein Vision and Language make it possible to streamline your workflows across sales, service, and marketing so that you can do things like: visual product search, product identification, intelligent case routing, and automated planogram analysis. ## December 6, 2017 ### NEW #### Add feedback to object detection models. If your object detection model misclassifies images, you can use the feedback API to add those images, along with their correct labels, to the dataset. After you add feedback to the dataset you can: - Train the dataset to create a new model - Retrain the dataset to update the model and keep the same model ID See [Add Feedback to a Dataset] and [Create Feedback Examples From a Zip File]. ### CHANGED #### Model training must be complete before you can delete a dataset. If a dataset is being trained and has an associated model with a status of `QUEUED` or `RUNNING`, you must wait until the training is complete before you can delete the dataset. ## October 27, 2017 ### CHANGED #### JWT token is now longer. The JWT tokens you use to call the API are now longer. You see this change whether you use the token [web page](https://api.einstein.ai/token) to get a token or whether you generate the token in code by calling the `/oauth2/token` endpoint. See [Generate an OAuth Token](/docs/analytics/einstein-vision-language/references). ### NEW #### Get learning curve metrics for Einstein Language models. Use this new API call to get the model metrics for each epoch (training iteration) performed to create a sentiment or intent model. See [Get Model Learning Curve](/docs/analytics/einstein-vision-language/references). #### Use the precision-recall curve metrics to understand your Einstein Language model. When you get the model metrics, the API now returns the precision-recall curve for your model. These metrics help you understand how well the model performs. See [Get Model Metrics](/docs/analytics/einstein-vision-language/references). ## October 16, 2017 ### NEW #### Einstein Object Detection now available. Use this API to train models to recognize and count multiple distinct objects within an image. This API is part of Einstein Vision, so you use the same calls as you do for image and multi-label models. But the data you use to create the models is different. See [Create a Dataset From a Zip File Asynchronously]. #### New Trailhead module: Einstein Intent API Basics. Build a deep-learning custom model to categorize text and automate business processes. See [Einstein Intent API Basics](https://trailhead.salesforce.com/modules/einstein_intent_basics). ## October 3, 2017 ### NEW #### Get all examples for a label. You can now return all examples for a single label by passing in the label ID. This API call is available in both Einstein Vision and Einstein Language. For Einstein Vision, see [Get All Examples for Label]. For Einstein Language, see [Get All Examples for Label]. ## July 31, 2017 ### NEW #### Pass parameters as JSON when classifying text using the Einstein Language APIs. You can now pass text in JSON when calling the `/intent` and `/sentiment` resources. See [Prediction for Intent](/docs/analytics/einstein-vision-language/references) and [Prediction for Sentiment](/docs/analytics/einstein-vision-language/references). ## July 27, 2017 ### CHANGED #### Einstein Image Classification API limits updated. - The image file name maximum length increased from 100 to 150 characters. - There's no longer a maximum number of examples you can create using the [Create an Example] call. #### Add single examples to a dataset. You can use the [Create an Example] call to add an example to a dataset that was created from a .zip file. #### Unicode characters now supported in all APIs. These elements can now contain unicode characters: - .zip file name - directory or label name - file or example name - dataset name #### Default split ratio changed. In the Einstein Language APIs, the default split ratio used during training is now 0.8. With this split ratio, 80% of the data is used to create the model and 20% is used to test the model. #### The minimum number of examples changed in the Einstein Language APIs. - A dataset with a type of `text-intent` must have at least five examples per label. - A dataset with a type of `text-sentiment` must have at least five examples per label. ## June 28, 2017 ### NEW #### Einstein Language (Beta) released. Einstein Language includes two APIs that you can use to unlock powerful insights within text. - Einstein Intent (Beta)—Categorize unstructured text into user-defined labels to better understand what users are trying to accomplish. - Einstein Sentiment (Beta)—Classify the sentiment of text into positive, negative, and neutral classes. See [Introduction to Salesforce Einstein Language](https://developer.salesforce.com/docs/analytics/einstein-vision-language/Einstein-Language/einstein-language.md). ## June 27, 2017 ### NEW #### Einstein Image Classification API version 2.0 released. This table lists all the changes to the API in the new version. Einstein Vision is now the umbrella term for all of the image recognition APIs. The Einstein Vision API is now called the Image Classification API. Use the version selector at the top of this page to switch to the documentation for another version. #### The API now uses the https://api.einstein.ai endpoint. When you access the Einstein Vision and Language APIs, you can now use this new endpoint. For example, the endpoint to get a dataset is `https://api.einstein.ai/v2/vision/datasets/`. The old api.metamind.io endpoint still works, but be sure to update your code to use the new endpoint. #### Optimize your model using feedback. Use the feedback API to add a misclassified image with the correct label to the dataset from which the model was created. - Use the new API call to add a feedback example. See [Create a Feedback Example]. - The call to get all examples now has three new query parameters: `feedback`, `upload`, and `all`. Use these query parameters to refine the examples that are returned. See [Get All Examples]. - The call to train a dataset and create a model now takes the `trainParams` object `{"withFeedback": true}`. This option specifies that the feedback examples are used during the training process. By default, the feedback examples aren't used during training if you don't specify this value. See [Train a Dataset]. #### Retrain a dataset and keep the same model ID. There's now a call to retrain a dataset, for example, if you added new data to the dataset or you want to include feedback data. Retraining a dataset lets you maintain the model ID which is ideal if you reference the model in production code. See [Retrain a Dataset]. #### Multi-label datasets are available. The new dataset type `image-multi-label` enables you to specify that the dataset contains multi-label data. Any models you create from this dataset have a `modelType` of `image-multi-label`. See [Determine the Model Type You Need]. #### There are two new calls to get the model metrics and the learning curve for a multi-label model. See [Get Multi-Label Model Metrics] and [Get Multi-Label Model Learning Curve]. #### Get up and running with multi-label predictions using our prebuilt multi-label model. This multi-label model is used to classify a variety of objects. #### Use the `numResults` parameter to limit prediction results. The `numResults` optional request parameter lets you specify the number of labels and probabilities to return when sending in data for prediction. This parameter can be used with both Einstein Vision and Einstein Language. #### Use global datasets to include additional data in your model. Global datasets are public datasets that Salesforce provides. When you train a dataset to create a model, you can include the data from a global dataset. One way you can use global datasets is to create a negative class in your model. See [Use Global Datasets](https://developer.salesforce.com/docs/analytics/einstein-vision-language/Best-Practices/use-global-datasets.md). ### CHANGED #### Dataset type is required when you create a dataset. When you call the API to create a dataset, you must pass in the `type` request parameter to specify the type of dataset. Valid values are: - `image`—Standard classification dataset. Returns the single class into which an image falls. - `image-multi-label`—Multi-label classification dataset. Returns multiple classes into which an image falls. #### Getting all datasets returns a maximum of 25 datasets. If you omit the `count` parameter, the call to get all datasets returns 25. If you set the `count` query parameter to a value greater than 25, the call returns 25 datasets. See [Get All Datasets](/docs/analytics/einstein-vision-language/references). ### DEPRECATED The following calls have been removed from the Einstein Image Classification API in version 2.0. - Create a label. You must pass in the labels when you create the dataset. `/vision/datasets//labels` - Get a label. `vision/datasets//labels/` - Get an example. `/vision/datasets//examples/` - Delete an example. `/vision/datasets//examples/` # Reset Your Private Key After you sign up for an account, you download or save your private key in the form of a .pem file. But sometimes things happen. If you lose your private key, you can reset it. Resetting your key generates a new key, but you still have access to your datasets and models. However, your previous key will no longer work. 1. Navigate to the reset page at [https://api.einstein.ai/web/reset](https://api.einstein.ai/web/reset). 2. Type the email address associated with your Einstein Vision and Language account. Note that this is the account email address and not your Salesforce username. 3. Click **Reset My Private Key**. ![ResetPrivateKey](https://developer.salesforce.com/media/reset_private_key.png) 4. You'll receive an email with a link that takes you to a page where you can download your key. :::caution The **Download Key** button is only supported in the most recent version of Google Chrome TM. If you're using a different browser, you can cut and paste your key into a text file and save it as `einstein_platform.pem`. ::: # Select the Best Use Case Now that you understand more about deep learning and the development cycle, it’s time to investigate how you can use Einstein Vision and Einstein Language to solve your own business problems. Here’s a list of considerations when selecting a successful use case. - **Select the right business problem.** During the process of identifying a good use case, there are various factors to consider. First, find business problems in your organization that fit the technologies available in Einstein Vision or Einstein Language. Look for tasks that can be automated and are a good match for these services. Be sure that the problem you solve isn’t too big or complex. This way you can achieve success more quickly. And consider whether solving the problem has value to your organization. - **Select a use case for which you have data.** Data is central to successful deep learning projects. When you identify a high-value business problem, confirm that you have enough data to solve the problem using deep learning. The quality of the data is also important. For Einstein Language, do you have enough sentiment or intent text data for each label? - **Understand the effort required for data processing (labeling or formatting data).** After you identify that you have enough high-quality data, be prepared for the time and effort required to process the data. The data must be sorted and labeled. If text data is in long paragraphs, you need to process it so that individual phrases can be extracted and labeled. - **Define the business success metrics.** When you implement deep learning, your goal isn’t to use AI; the goal is to solve a business problem. To ensure that your solution is working, define what success looks like from the business perspective. For example, if you’re implementing a deep learning project to route incoming support cases, define up front how many cases you want to route correctly and in what timeframe. If your solution correctly routes a case, identify the time and cost savings. Understand the impact that the solution has on your business. - **Bring in the right expertise.** To build and implement deep learning solutions requires people in your organization that understand AI and deep learning. You can use Einstein Vision and Language managed services to quickly build solutions. But it’s still necessary to have people that understand the deep learning life cycle and technologies. Be sure that you have people that can frame the problem, collect the right data, label and process the data, understand the model, interpret the model metrics, and understand how to use feedback to improve the model. ## Resources - Blog post: [How to Choose Your First AI Project](https://hbr.org/2019/02/how-to-choose-your-first-ai-project) # Set Up Authorization The Einstein Vision and Language APIs use OAuth 2.0 JWT bearer token flow for authorization. Use the [token page](https://api.einstein.ai/web/token) to upload your key file and generate a JWT token. 1. Type your email address or account ID. - If you signed up using Salesforce, use the email address associated with your user in the Salesforce org you logged in to when you signed up. - If you signed up using Heroku, use the account ID contained in the `EINSTEIN_VISION_ACCOUNT_ID` config variable. 2. Click **Browse** and navigate to the `einstein_platform.pem` file that you downloaded when you signed up for an account. **tip:** The key file was previously named `predictive_services.pem`. If you signed up at an earlier time and you can't file your key file, try searching for a file by this name. 3. Set the number of minutes after which the token expires. 4. Click **Get Token**. You can now cut and paste the JWT token from the page. ![TokenPageWithToken](https://developer.salesforce.com/media/token_page_with_token.png) This page gives you a quick way to generate a token. In your app, you'll need to add the code that creates an assertion and then calls the API to generate a token. See [Generate an OAuth Token](/docs/analytics/einstein-vision-language/references). **tip:** The token you create when you use this site doesn't automatically refresh. Your application must refresh the token based on the expiration time that you set when you create it. # Support ## Check the Troubleshooting Resources - [API Error Codes and Messages](/docs/analytics/einstein-vision-language/references) - [Reset Your Private Key](https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/reset-your-private-key.md) ## Ask a Question on the Developer Forums If you need help working with the Einstein Vision and Language APIs, go to the [Einstein Platform developer forum](https://developer.salesforce.com/forums?communityId=09aF00000004HMGIA2#!/feedtype=RECENT&dc=Predictive_Services&criteria=ALLQUESTIONS) on Salesforce Developers. # Supported Languages Einstein Vision and Einstein Language support English by default. You can implement Einstein Intent in additional languages. Einstein Intent supports multiple languages. You still call the same APIs to create a dataset and train it to create a model. You specify the language of the dataset by passing in an optional `language` parameter. If you omit the parameter, the default is English (`en_US`). The Einstein NER (beta) model also supports unstructured text in these languages. ## Supported Languages | Language | Parameter value | | --------------------- | --------------- | | Brazilian Portuguese | `pt_BR` | | Chinese (Simplified) | `zh_CN` | | Chinese (Traditional) | `zh_TW` | | Danish (beta) | `da` | | Dutch | `nl_NL` | | English (UK) | `en_GB` | | English (US) | `en_US` | | French | `fr` | | German | `de` | | Italian | `it` | | Japanese | `ja` | | Korean (beta) | `ko` | | Portuguese | `pt_PT` | | Russian | `ru` | | Spanish | `es` | | Swedish (beta) | `sv` | **note:** As a beta feature, Danish, Korean, and Swedish language support is a preview and isn’t part of the “Services” under your Main Services Agreement with Salesforce. Use this feature at your sole discretion, and make your purchase decisions only on the basis of generally available products and features. Salesforce doesn’t guarantee general availability of this feature within any particular time frame or at all, and we can discontinue it at any time. This feature is for evaluation purposes only, not for production use. It’s offered as is and isn’t supported, and Salesforce has no liability for any harm or damage arising out of or in connection with it. All restrictions, Salesforce reservation of rights, obligations concerning the Services, and terms for related Non-Salesforce Applications and Content apply equally to your use of this feature. # Terms of Service If you’re signing up with your Salesforce (“SFDC”) or Heroku credentials: Your use of the Einstein.ai Service(s) is subject to the following [Main Services Agreement](https://www.salesforce.com/content/dam/web/en_us/www/documents/legal/Salesforce_MSA.pdf) (“MSA”), as well as to the following Supplemental Terms, which are hereby incorporated into such MSA by reference. Capitalized terms not defined herein shall have the meaning ascribed to them in the MSA. ## Supplemental Terms: "Einstein.ai API” means the API endpoint accessible at [https://api.einstein.ai](https://api.einstein.ai). "Einstein.ai Service(s)” means the Einstein.ai API for the Einstein Vision Service, or any other associated online service you access via the Einstein.ai API. "Prediction” means an API call to the Einstein.ai Service(s) with the purpose of generating a prediction. “Beta Services” means SFDC services or functionality that may be made available to Customer to try at its option at no additional charge which is clearly designated as beta, pilot, limited release, developer preview, non-production, evaluation, or by a similar description. Your use of the Einstein.ai Service(s) includes 1,000 free Predictions per calendar month. SFDC reserves the right to limit the size of databases, the number of Models that You can train and Your general access to the Einstein.ai API. Unused Predictions are forfeited at the end of each calendar month and do not roll over to subsequent calendar months. The beginning and end of each calendar month will conform with UTC Time.

Einstein.ai Service(s) are available in English only. You acknowledge that SFDC may access Your Data submitted to the Einstein.ai Service(s) for the purpose of developing and improving the Einstein.ai Service(s), and any other of SFDC’s current and future similar or related features, products and/or services. You shall comply with all applicable laws and regulations in Your collection, use, processing and retention of any images, text, sounds and/or other data submitted to the Einstein.ai Service(s), and hereby represent and warrant that You have obtained all consents and provided all notices (including, if applicable, through clear disclosure in privacy policies) that are necessary to use, process, and retain all images, text, sounds and/or other data You submit to the Einstein.ai Service(s). Sensitive personal data may not be submitted to the Einstein.ai Service(s), including images, text, sounds or other data containing or revealing government-issued identification numbers; financial information (such as credit or debit card numbers, any related security codes or passwords, and bank account numbers); racial or ethnic origin, political opinions, religious or philosophical beliefs, trade-union membership, information concerning health or sex life; information related to an individual’s physical or mental health; and information related to the provision or payment of health care. Additionally, You may not use the Einstein.ai Service(s) to create or analyze biometric identifiers such as face prints or fingerprints, or scans of eyes, hands or facial geometry, nor may You use the Einstein.ai Service(s) for the purposes of analyzing, profiling or targeting someone’s racial or ethnic origin, political opinions, religious or philosophical beliefs, trade union membership, age, gender, sex life, sexual orientation, criminal convictions, disability, health status or medical condition. We may update these Einstein.ai API Terms of Service at any time. We’ll post notice of modifications to these terms on this page. We may also notify you of material revisions, for example via a service notification or an email to the email associated with your Salesforce or Heroku account. Your continued use of the Einstein.ai Service(s) means you agree to the updated terms, and if you don’t agree to the updated terms, you must stop using the Einstein.ai API. Effective June 28, 2017 # Use Feedback to Improve a Model **important:** As of August 14, 2023 the withFeedback attribute has been deprecated and will no longer be supported by Salesforce. To append data to a dataset, add the data locally and create a new dataset and train a new model. Implementing a deep learning model is an iterative process. Continuing to refine your production model is part of the life cycle. When you put a model into production, it’s a good idea to let users identify misclassified data as they send in data and get predictions. Creating a method to track misclassified data means that you can quickly get a model up and running. You can then continue to improve the model as you learn more about how it’s used and performs. Implement a feedback loop in your apps with the feedback APIs. The high-level process includes these steps. 1. Build functionality in your app to identify text data that was misclassified. 2. Add the data, along with the correct label, to the dataset. Note that you add the feedback text to the dataset from which the model was created. 3. Train or retrain the dataset, and use the `withFeedback` training parameter. 4. Use the new or updated model after the feedback is incorporated. # Use Global Datasets Global datasets are public datasets that Salesforce provides. You can use these datasets to include additional data during training when you create a model. One way you use global datasets is to create a negative class in your model. A negative class is returned in a prediction for an image that doesn't match any of the other classes. For example, the beaches and mountains model has only two classes: Beaches and Mountains. If you classify an image of an airplane using that model, the response returns probabilities for those two classes, which aren’t very helpful. Instead, you can use the global dataset to add another label named "other". Now when you train the beaches and mountains dataset, you include the global dataset. Your original dataset stays the same, but the data from the global dataset is added to the model. The model then contains these labels: Mountains, Beaches, and other. ## Use a Global Dataset to Create a Model This cURL command trains a dataset and creates a model using one of the global datasets. Replace with your dataset ID. The global dataset ID specified here (1005161) is the ID for a standard image classification dataset. ```cURL curl -X POST -H "Authorization: Bearer " -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Beaches and Mountains w/Global Other Dataset" -F "datasetId=" -F "trainParams={\"withGlobalDatasetId\" : 1005161}" https://api.einstein.ai/v2/vision/train ``` Now when you send an image of an airplane for classification, the model returns this response. ```json { "probabilities": [ { "label": "other", "probability": 0.99999857 }, { "label": "Beaches", "probability": 8.929781e-7 }, { "label": "Mountains", "probability": 5.91046e-7 } ], "object": "predictresponse" } ``` You can use global datasets in both train and retrain API calls. ## Find Out What Global Datasets are Available Use the `global` query parameter to return a list of global datasets. This cURL command returns all the global datasets. ```cURL curl -X GET -H "Authorization: Bearer " -H "Cache-Control: no-cache" https://api.einstein.ai/v2/vision/datasets?global=true ``` This cURL call returns a response similar to this one. Note the dataset type, and be sure that the global dataset type matches the type of the dataset that you train to create a model. ```json { "object": "list", "data": [ { "id": 1005161, "name": "other", "createdAt": "2017-06-27T23:21:16.000+0000", "updatedAt": "2017-06-27T23:21:19.000+0000", "labelSummary": { "labels": [ { "id": 24197, "datasetId": 1005161, "name": "other", "numExamples": 455 } ] }, "totalExamples": 455, "totalLabels": 1, "available": true, "statusMsg": "SUCCEEDED", "type": "image", "object": "dataset" } ] } ``` Global datasets are a little different than datasets that you create using your own data. - You can return global datasets only by using the API call that gets all datasets. - Global datasets can only be used together with one of your own datasets to create a model. - You can’t get a single global dataset, get its examples, train it, or delete it. # Use the OCR Model to Detect Text In this walkthrough, you use the Einstein OCR API to pass an image into the OCR model. The model detects any text in the image and returns the text and other details in the response. ## Step 1: Get a Token A valid JWT token is required to make API calls. 1. Find your key file. If you already created an account, locate the `einstein_platform.pem` file that you downloaded as part of that process. This file contains your private key. 2. Install cURL. You use the cURL command-line tool throughout the following walkthrough. The tool is installed by default on Linux and OSX. If you don’t already have it installed, download it from [https://curl.haxx.se/download.html](https://curl.haxx.se/download.html). Alternatively, you can use Postman to make API calls. 3. Get a token. The Einstein Intent API uses the OAuth 2.0 JWT bearer token flow for authorization. Use the [token page](https://api.einstein.ai/token) to upload your key file and generate a JWT token. ## Step 2: Call the OCR Model Copy and paste the following cURL call into a text editor and replace `` with your valid token. This command works on Windows. If you're on Linux, it can require some format changes, such as from double quotes to single quotes. This command uses the model OCRModel. ```cURL curl -X POST -H "Authorization: Bearer " -F sampleLocation="https://www.publicdomainpictures.net/pictures/240000/velka/emergency-evacuation-route-signpost.jpg" -F modelId="OCRModel" https://api.einstein.ai/v2/vision/ocr ``` The URL references this image. ![Signpost](https://developer.salesforce.com/media/emergency-evacuation-route-signpost.jpeg) You get a response like the following JSON. ```json { "object": "predictresponse", "probabilities": [ { "probability": 0.99937266, "label": "ROUTE", "boundingBox": { "minX": 588, "minY": 692, "maxX": 1145, "maxY": 807 }, "attributes": { "pageNumber": "1" } }, { "probability": 0.99471515, "label": "EMERGENCY", "boundingBox": { "minX": 355, "minY": 217, "maxX": 1370, "maxY": 328 }, "attributes": { "pageNumber": "1" } }, { "probability": 0.99469215, "label": "EVACUATION", "boundingBox": { "minX": 345, "minY": 433, "maxX": 1376, "maxY": 567 }, "attributes": { "pageNumber": "1" } } ], "task": "text" } ``` ![EmergencyEvacuationRoute](https://developer.salesforce.com/media/emergency-evacuation-route-signpost_ocr.jpg) # What is Einstein NER (Beta)? NER stands for named entity recognition. The NER model takes unstructured text data and identifies entities within that text. **note:** As a beta feature, Einstein NER is a preview and isn’t part of the “Services” under your Main Services Agreement with Salesforce. Use this feature at your sole discretion, and make your purchase decisions only on the basis of generally available products and features. Salesforce doesn’t guarantee general availability of this feature within any particular time frame or at all, and we can discontinue it at any time. This feature is for evaluation purposes only, not for production use. It’s offered as is and isn’t supported, and Salesforce has no liability for any harm or damage arising out of or in connection with it. All restrictions, Salesforce reservation of rights, obligations concerning the Services, and terms for related Non-Salesforce Applications and Content apply equally to your use of this feature. If you send in this string: `Marc Benioff, the CEO of Salesforce, gave the keynote speech at the conference in Paris last week.` The model returns a prediction result like this JSON. ```json { "probabilities": [ { "probability": 0.99962294, "label": "PERSON", "token": "Marc Benioff", "span": { "start": 0, "end": 12 }, "normalizedData": [ { "firstName": "Marc", "lastName": "Benioff", "middleName": "", "suffix": "", "salutation": "", "type": "person" } ] }, { "probability": 0.9115802, "label": "ORGANIZATION", "token": "Salesforce", "span": { "start": 27, "end": 37 }, "normalizedData": [] }, { "probability": 0.9537982, "label": "DATETIME", "token": "20 years ago", "span": { "start": 62, "end": 74 }, "normalizedData": [ { "value": "2000-08-01T00:00:00.000-07:00", "unit": "month", "type": "value" } ] } ], "object": "predictresponse" } ``` ![NERModelOverview](https://developer.salesforce.com/media/ner_model_overview.png) The model identifies a sequence of words (the token) that are categorized as a particular entity. The model returns the type of entity it identifies and the probability that the token is that entity. For example, the words "Marc Benioff" are the token and the entity returned by the model is PERSON. In this case, the model identifies four entities in the text: - a person (Marc Benioff) - an organization (Salesforce) - a location (Paris) - a date/time (last week) ## Entities Supported Here are the entities that the NER model identifies. - DATETIME - DURATION - EMAIL - LOCATION - MONEY - NUMBER - ORGANIZATION - PERCENT - PERSON - PHONE-NUMBER - URL For more information about the NER endpoint, see [Detect Entities in Text](/docs/analytics/einstein-vision-language/references). # What is Einstein OCR? Einstein OCR (optical character recognition) provides models that detect typed and handwritten text in an image or PDF. You access the models from a single REST API endpoint. Each model is used for specific use cases, such as business card scanning, product lookup, and digitizing documents and tables. | Model ID | Used For | Recommended Task Parameter Values | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `OCRModel` |
  • Business card, pass a `task` parameter value of `contact`
|
  • `contact`
| | `tabulatev2` |
  • Unformatted text, pass a `task` parameter value of `text`.
  • Text in tables. Pass a task `parameter` value of `table`.
  • Standard form, pass a `task` parameter value of `form`.
  • Custom form, pass a `task` parameter value of `form` .
  • Invoices, pass a `task` parameter value of invoice.
  • Support for typed and handwritten text recognition.
|
  • `text`
  • `table`
  • `form`
  • `invoice`
| When you call the API, you send in an image, and the JSON response contains various elements based on the value of the task parameter: - String of alphanumeric characters that the model predicts. - Confidence (probability) that the detected bounding box contains text. - XY coordinates for the location of the character string within the image (also called a bounding box). - For tabular data, the table row and column in which the text is located. - For business cards, the entity type of the detected text such as ORG, PERSON, and so on. - For common forms, key-value pairs that contain the detected text, bounding box data for the detected text, and the entity (a consistent field name). Here’s a sample image. The orange boxes and text indicate the detected text. **note:** Currently, Einstein OCR supports only English.