--- schema_version: "1" id: aae48e41b05d8c316c4de210af8b5eddfcb98274393c63477bd9538b9ed8def4 title: Creating Model Proxies collection: mulesoft version: current locale: en-us portal_url: https://docs.salesforce.com/mulesoft/current/en-us/general/model-proxy-create-model-proxy.md canonical_url: https://docs.mulesoft.com/general/model-proxy-create-model-proxy source_site: docs.mulesoft.com content_type: documentation content_hash: 0ac8a8f8a81b6c970f41b4410dd498693a8f54b80e2545fd3b28d7beb49c5ee7 description: MuleSoft Documentation Site latest: true --- --- product: Getting Started --- # Creating Model Proxies > For the full documentation index, see: https://docs.mulesoft.com/llms.txt You can configure a model proxy to route LLM traffic across different models and providers using model-based or semantic routing strategies. When you create a model proxy, you define the endpoint format, routing strategy, and at least one route that maps requests to a supported LLM provider and model. After deployment to Omni Gateway, you can edit the proxy configuration at any time. > [!NOTE] A large Omni Gateway supports up to 50 Model Proxies. > [!NOTE] Automate model proxy provisioning using the MuleSoft MCP Platform Server. For details, see > [provision\_mcp\_server](https://docs.mulesoft.com/mulesoft-mcp-server/reference-mcp-tools-platform#provision-mcp-server) > in the MuleSoft MCP Platform Server Tool Reference. ## Before You Begin Before getting started, make sure you have: - An Anypoint Platform account - API Manager: API Creator permission For more information, see [Enhanced Experience Permissions](https://docs.mulesoft.com/general/exp-home-start#permissions). - A deployed Omni Gateway version 1.11.4 See [Deploy a Managed Omni Gateway](https://docs.mulesoft.com/gateway/latest/gateway-managed-set-up). - API keys to authenticate with your LLM Providers. - A [configured semantic routing service](https://docs.mulesoft.com/general/model-proxy-semantic-service) if you want to use semantic routing. - A [configured semantic caching service](https://docs.mulesoft.com/general/model-proxy-semantic-caching-service) if you want to enable semantic caching. - A connection to an external vault (AWS Secrets Manager, Microsoft Azure Key Vault, or HashiCorp Vault) configured under **Platform** > **Providers** if you want to authenticate a route with a secret from your vault. For more information, see [Using Credentials Stored in External Vaults](https://docs.mulesoft.com/general/exp-vaults-manage). ## Create a Model Proxy 1. In the navigation pane, select **Model Proxies**. 2. Click **Add Model Proxy** 3. Configure the proxy parameters: 1. **Proxy Name**: Define a name for the Model Proxy. 2. Enter a **Description** of what this proxy does (up to 256 characters). 3. **Base Path**: Define a base path for the proxy endpoint (for example: `/billing-ai`). 4. **Format**: Select an endpoint format: - **OpenAI**: Select the OpenAI API format to send requests to all supported LLM providers (including Gemini and Anthropic). Supports multi-routing and fallback mechanisms. You can’t change this format later. - **Anthropic**: Select the Anthropic API format for native Anthropic Claude model requests. Supports model-based routing only. You can’t change this format later. - **Gemini**: Select the Gemini API format for native Google Gemini model requests. Supports model-based routing only. You can’t change this format later. 5. **Environment**: Select the environment for the Model Proxy. 6. **Omni Gateway**: Select an Omni Gateway to deploy the Model Proxy to. 7. Enter a **Consumer Endpoint**: the URL where the Model Proxy is accessible (for example, `[https://api.example.com](https://api.example.com)`). 8. **Port**: Enter the port number for the Model Proxy (for example, `8081`). This field is required. 9. Configure **TLS Configuration** to secure the proxy endpoint: - **Secret Group**: Select the secret group that contains your TLS context. If you don’t see your secret group, ensure that it is downloadable. For more information, see [Add a TLS Context](https://docs.mulesoft.com/gateway/latest/conn-tls-config#add-a-tls-context). - **TLS Context**: Select the TLS context to use. If you don’t see your TLS context, click **Configure a new TLS Context** to create one. 4. Click **Continue**. 5. Configure the routing strategy: 1. Select a routing strategy: - **Model-based**: Route requests based on the model specified in the request payload. - **Semantic**: Route requests based on semantic analysis of the prompt content. Requires a configured semantic service. Not supported for the Anthropic or Gemini endpoint formats. 6. Configure at least one route: 1. Provide a route name (for example, `Route A`). 2. Click **Add headers** to add routing headers that filter requests by region, SLA, or custom rules. For example, `Region: US` 3. Configure the target: - **Provider**: Select your LLM provider from these options: - **OpenAI** - **Gemini** - **Azure OpenAI** - **Bedrock OpenAI** - **Bedrock Anthropic** - **NVIDIA** - **Anthropic** - **xAI** - **Mistral** - Any custom provider published to Anypoint Exchange. For more information, see [Publishing Custom LLM Providers to Anypoint Exchange](https://docs.mulesoft.com/general/model-proxy-publish-custom-llm-metadata). - **Request model override**: Select a target model to override the model version specified in the payload. Selecting **Use model from request** sends the request to the model specified in the request. A target model is required for semantic routing. - **Destination URL**: Specify the URL for your provider endpoint. Ensure the URL is correct and edit if necessary. 4. Under **TLS Configuration**, click **Add TLS Context** to secure the connection to the provider endpoint with a TLS context. 5. Configure authentication: - **API Key**: Choose one of the following: - **Static**: Use your provider API key. Enter a static API key for the provider endpoint. - **Dynamic**: Retrieve your key dynamically from the request. Define a DataWeave script to extract the API key from the incoming request. - **From vault**: Use a secret stored in an external vault. Select a secret group, then select a secret from that group. For more information, see [Using Credentials Stored in External Vaults](https://docs.mulesoft.com/general/exp-vaults-manage). 7. Click **Add Route** to add additional routes. Complete the previous steps to configure each new route 8. Expand **Advanced Options** and enable **Enable semantic caching** to cache and reuse LLM responses based on semantic similarity. Select a configured semantic caching service from the list. For more information, see [Configuring Semantic Caching Services](https://docs.mulesoft.com/general/model-proxy-semantic-caching-service). 9. Click **Add Model Proxy**. ## Edit a Model Proxy To edit a Model Proxy: 1. In the navigation pane, select **Model Proxies**. 2. Click the name of the Model Proxy you want to edit. 3. Click **Edit Configuration**. 4. Make the necessary edits. 5. Click **Save Changes**.