--- schema_version: "1" id: 7af8c0b589067ba580b7ced82eddc1fdecd3128a55575609eb5cb46ed4209855 title: Generate a Robots.txt File with Business Manager for B2C Commerce collection: admin version: current locale: en-us portal_url: https://docs.salesforce.com/admin/current/en-us/cc/264-0-0/b2c_generating_a_robots_txt_file.md canonical_url: https://help.salesforce.com/s/articleView?id=cc.b2c_generating_a_robots_txt_file.htm&release=264.0.0&type=5 source_site: help.salesforce.com content_type: documentation content_hash: 1eaeb92ef58993f9be10506356b10efb360886e7dadcf0a6562d6ef1a8a79e76 description: To create a robots.txt for one or more sites individually, use Business Manager. This robots.txt file is served to any requesting crawlers from the application server. It's stored as a site preference and can be replicated from one instance to another. products: - Commerce - B2C Commerce release: "264" --- # Generate a Robots.txt File with Business Manager for B2C Commerce To create a robots.txt for one or more sites individually, use Business Manager. This robots.txt file is served to any requesting crawlers from the application server. It's stored as a site preference and can be replicated from one instance to another. When you select **Custom robots.txt definition**, the sample text includes `User-agent` entries for both traditional search engine crawlers and AI crawlers. For information about which crawlers request your robots.txt file, see [Robots.txt Files for B2C Commerce](https://help.salesforce.com/s/articleView?id=cc.b2c_using_the_robots_txt_file_to_prevent_crawling_of_development_and_staging_instances.htm&release=264.0.0&type=5 "When a robot visits a website, such as http://www.yourshophere.com/, it first checks for the robots.txt file and analyzes it to determine if it can index the site. For the robots.txt file to be used by Salesforce B2C Commerce, you must set up your hostname alias."). For the most current list of AI crawler user-agent tokens, see the documentation published by each provider. - You can write up to 50,000 characters to this file in Business Manager. - URIs are case-sensitive, and "/robots.txt" string must be all lower-case. - Blank lines aren’t permitted within a single record in the "robots.txt" file. - There must be exactly one `User-agent` field per record. The robot should be liberal in interpreting this field. - A case-insensitive substring match of the name without version information is recommended. - If the value is "\*", the record describes the default access policy for any robot that hasn’t matched any of the other records. - It isn't allowed to have multiple such records in the "/robots.txt" file. - The "Disallow" field specifies a partial URI that isn't to be visited. This can be a full path, or a partial path; any URI that starts with this value won’t be retrieved. For example, ``` Disallow: /help ``` disallows both /help.html and /help/index.html, whereas ``` Disallow: /help/ ``` would disallow /help/index.html but allow /help.html. An empty value for _Disallow,_ indicates that all URIs can be retrieved. - At least one `Disallow` field must be present in the robots.txt file. 1. Click App Launcher ![App Launcher](https://sf-zdocs-cdn-prod.zoominsoftware.com/tdta-cc-b2c_merchandiser_administrator-264-0-0-production-enus/ea76f372-6cfa-42a6-8e7c-05c6df7abed8/cc/b2c_merchandiser_administrator/images/appLauncher.png), and then select **Merchant Tools** | **Site** | **SEO & Discoverability** | **Robots** 2. Select the instance type to create a robots.txt file. If you want to create a robots.txt file for a Production instance, you can do so on a Staging instance and replicate the site preferences, where the robots.txt file definition is stored, from the Staging instance to the Production instance. 3. Select one of these options: 1. **Use the robots.txt file from a deployed cartridge**: Use Google Search Console or another third-party tool to generate your robots.txt file. Add the file to a cartridge on your site path. There can only be one robots.txt file per site. If you want to generate a robots.txt file using another tool and upload it to your cartridge. This option is most useful if you want to use the same robots.txt file for multiple sites. This is not recommended, because usually you want to have different settings for different instance types. For example, you don't want your sandbox or staging sites to be crawled, but you do want your production sites to be crawled. This can cause issues when replicating code to production. This option is only selected before a site goes live to test the robots.txt file. 2. **Define an instance type-specific robots.txt** (recommended): Use this option to have B2C Commerce generate a robots.txt file for you or specify a custom robots.txt file for each of your instances. 4. If you selected **Define an instance type-specific robots.txt**, select one of these options: 1. **All spiders are allowed to access any static resources** (recommended for Production): Use this if you want your storefront to be crawled and available to external search engines, such as Google. This generates a site-specific robots.txt file that indicates spiders can crawl the static resources for the site. 2. **All spiders are disallowed to access any static resources** (recommended for Staging): Use this if you don't want your storefront to be crawled and available to external search engines, such as Google. This generates a site-specific robots.txt file that indicates to spiders that they shouldn't crawl the static resources for the site. 3. **Custom robots.txt definition**: Use this option if you want to control which parts of your storefront are crawled and available to search engines and AI crawlers. When you select this option, Business Manager populates the text area with a sample robots.txt file. All entries in the sample are commented out. Uncomment and customize the entries for your site. The following sample text was introduced in B2C Commerce 26.1 and includes user-agent entries for AI crawlers: ``` ################################################################################ ## SAMPLE robots.txt - Uncomment and customize as needed for your site ################################################################################ ## To allow access to all crawlers, including AI crawlers, use a general rule # User-agent: * ## To allow or deny access to specific crawlers, enable them individually # User-agent: Googlebot # User-agent: OAI-SearchBot # User-agent: GPTBot # User-Agent: ChatGPT-User # User-Agent: ClaudeBot # User-Agent: Claude-SearchBot # User-Agent: Claude-User # User-Agent: Perplexity-User # User-Agent: PerplexityBot ## Example rules to allow access to content pages, but not customer ## specific or functional pages: # Allow: / # Disallow: */account/ # Disallow: */cart/ # Disallow: */login/ # Disallow: */wishlist/ # Disallow: */Login-Show/ # Disallow: */Registration-Shopper/ # Disallow: */Checkout/ # Disallow: */Order/ # Disallow: */Order-History/ # Disallow: */Order-Confirmation/ # Disallow: */Order-Track/ ## A Crawl-delay can help throttle server impact. AI bots may not honor ## this, but this is good practice. # Crawl-delay: 10 ## To help search engines find your content, add a pointer to your sitemap. # Sitemap: https://www.yourdomain.com/sitemap.xml ``` If your storefront is built on Salesforce Reference Architecture (SFRA) or SiteGenesis, we recommend also adding the search refinement URL parameter directives shown in the postrequisites to your custom file. 5. Click **Apply**. 6. Select **Administration** | **Sites** | **Manage Sites** | **site** | **Cache**. 7. In the Static Content and Page Caches section, click **Invalidate**. For information on where to upload your robots.txt file, see [Upload a Robots.txt File.](https://help.salesforce.com/s/articleView?id=cc.b2c_uploading_your_robots_txt_file.htm&release=264.0.0&type=5 "If you choose to create a robots.txt file in an external file, you must upload the file to the correct location on your B2C Commerce server.") If your storefront is built on SFRA or SiteGenesis, we recommend adding the following lines to your robots.txt. These directives prevent crawlers from indexing filtered or sorted search result pages, which helps avoid duplicate content and keeps search indexes focused on your primary product and category pages. These parameters are specific to the SFRA and SiteGenesis URL format and don't apply to PWA Kit storefronts, which use Shopper Commerce API (SCAPI) query parameters such as `refine` and `sort`. ``` # Search refinement URL parameters (SFRA and SiteGenesis) Disallow: /*pmin* Disallow: /*pmax* Disallow: /*prefn1* Disallow: /*prefn2* Disallow: /*prefn3* Disallow: /*prefn4* Disallow: /*prefv1* Disallow: /*prefv2* Disallow: /*prefv3* Disallow: /*prefv4* Disallow: /*srule* ``` If your storefront is built on PWA Kit (Composable Storefront), we recommend adding the following lines instead. PWA Kit uses SCAPI query parameters (`refine`, `sort`, and `offset`) in its `/search` and `/category/` routes. ``` # Search refinement, sort, and pagination URL parameters (PWA Kit) Disallow: /search?*refine* Disallow: /category/*?*refine* Disallow: /search?*sort* Disallow: /category/*?*sort* Disallow: /search?*offset* Disallow: /category/*?*offset* ``` Set the Googlebot crawl rate to Low through Google Search Console, as Google ignores the crawl-delay directive in robots.txt, outlined in [https://support.google.com/webmasters/answer/48620?hl=en](https://support.google.com/webmasters/answer/48620?hl=en). Most AI crawlers also ignore the `Crawl-delay` directive. To manage crawl rates for AI bots, consult each provider's documentation.