---
schema_version: "1"
id: 0507418ee83113d1099a6e293254ccc82b9e001876e18bc9218566ee5432cac6
title: Create a Custom robots.txt File for Your Experience Builder Site
collection: admin
version: current
locale: en-us
portal_url: https://docs.salesforce.com/admin/current/en-us/experience/264-0-0/networks_seo_configure.md
canonical_url: https://help.salesforce.com/s/articleView?id=experience.networks_seo_configure.htm&release=264.0.0&type=5
source_site: help.salesforce.com
content_type: documentation
content_hash: 3dd0a57c320d457d4057f0dc531a2390322edba1a83d299996303a06c0d0b41e
description: The robots.txt file for your site is automatically generated. It allows search engines to crawl your entire site. However, if you want to tell search engines not to crawl certain pages of your public site, you can create a custom robots.txt file.
products:
  - Experiences
release: "264"
---
# Create a Custom robots.txt File for Your Experience Builder Site

The robots.txt file for your site is automatically generated. It allows search engines to crawl your entire site. However, if you want to tell search engines not to crawl certain pages of your public site, you can create a custom robots.txt file.

### Required Editions

<table class="slds-table slds-table_bordered slds-max-medium-table_stacked slds-m-bottom_small edition"><colgroup><col></colgroup><tbody><tr class=""><td class="slds-cell-wrap" style="vertical-align:top;">Available in: Salesforce Classic (<a href="https://help.salesforce.com/s/articleView?id=xcloud.overview_edition_lex_only.htm&amp;type=5&amp;language=en_US" target="_blank">not available in all orgs</a>) and Lightning Experience</td></tr><tr class=""><td class="slds-cell-wrap" style="vertical-align:top;">Available in: <strong>Enterprise</strong>, <strong>Performance</strong>, and <strong>Unlimited</strong> Editions</td></tr><tr class=""><td class="slds-cell-wrap" style="vertical-align:top;">Applies to: <a href="https://help.salesforce.com/s/articleView?id=experience.exp_cloud_basics_glossary.htm&amp;release=264.0.0&amp;type=5">LWR and Aura sites</a></td></tr></tbody></table>

<table class="slds-table slds-table_bordered slds-max-medium-table_stacked slds-m-bottom_small permission">
        <colgroup>
          <col style="width:50%">
          <col style="width:50%">
        </colgroup>
        <thead class="slds-text-title_caps ">
          <tr class="">
            <th scope="colgroup" class="slds-cell-wrap" colspan="2" style="text-align:center;vertical-align:top;" id="d73149e86">User Permissions
              Needed</th>
          </tr>
        </thead>
        <tbody>
          <tr class="">
            <td data-label="User Permissions Needed" class="slds-cell-wrap" style="vertical-align:top;" headers="d73149e86 ">To create a Visualforce page and robots.txt file for your site:</td>

            <td class="slds-cell-wrap" style="vertical-align:top;" headers="d73149e86 ">Create and Set Up Experiences</td>

          </tr>
        </tbody>
      </table>

Use a Visualforce page to create a custom robots.txt file. Then add the URLs of the pages that you want search engines to crawl to the Allow section, and add the URLs that you don’t want search engines to crawl to the Disallow section. A common way to use a custom robots.txt file is to disallow your entire site and then selectively allow search engines to crawl individual pages.

```
Disallow: /   # hides everything from ALL bots
Allow: /<path-prefix-1>/s  # add path you want to open to bots
Allow: /<path-prefix-2>/s  # add path you want to open to bots
Sitemap: http://<site_URL>/s/sitemap.xml
```

Pages whose URLs you add to the Disallow section can still appear in search results. For example, if an allowed web page points to a disallowed URL, a search engine can index the URL when it’s crawling the allowed web page. Learn more in [Google’s Search Central Documentation](https://developers.google.com/search/docs/crawling-indexing/robots/intro). To hide or block a public site page from search results [use the noindex meta tag](https://help.salesforce.com/s/articleView?id=experience.networks_seo_tags.htm&release=264.0.0&type=5 "Set the title, description, and head properties for your Experience Builder site pages to improve search results.").

![Tip](https://sf-zdocs-cdn-prod.zoominsoftware.com/tdta-experience-communities-264-0-0-production-enus/bf7df212-91a1-4fa3-a007-4d6f3e3b470b/images/icon_tip.png)

Tip If you have more than one site, each with a different custom domain subpath, they share a robots.txt file. Therefore, when you create a custom robots.txt file, ensure that your search engine crawling directives account for all pages in all of your sites.

1. In a text file, create a list of include (Allow) and exclude (Disallow) directives for the areas of your site that you want to expose to or hide from search engines. Also include the path to all sitemaps for the domain—for example, `https://<site_URL>/s/sitemap.xml`.
    
    Only relative URLs are valid for Allow and Disallow directives.
2. To create a Visualforce page, from Setup, in the Quick Find box, enter Visualforce Pages. Then select **Visualforce Pages**, and click **New**.
3. Give the custom file a name. You can even call it robots.
    
    ![Content Snapshot](https://sf-zdocs-cdn-prod.zoominsoftware.com/tdta-experience-communities-264-0-0-production-enus/bf7df212-91a1-4fa3-a007-4d6f3e3b470b/communities/images/seo_custom.png)
4. Select **Available for Lightning Experience...**.
5. Copy your list of indexing directives and the sitemap path from the text file to the Visualforce page. For example:
    
    ```
    <apex:page contentType="text/plain">
        User-agent: *
        Disallow: /   
        Allow: /s  
        Allow: /mycommunity/s  
        Sitemap: https://example.com/s/sitemap.xml
        Sitemap: https://example.com/mycommunity/s/sitemap.xml
    </apex:page>
    ```
6. Save your work.
    
    The Visualforce page is automatically saved at the root-level of the preferred domain.
7. From Experience Workspaces, click **Administration** | **Pages**, and then click **Go to Force.com**. Then, on the Site Details page, click **Edit**.
8. In the Site Robots.txt field, enter the name of the Visualforce page that you created, or click ![Lookup search icon](https://sf-zdocs-cdn-prod.zoominsoftware.com/tdta-experience-communities-264-0-0-production-enus/bf7df212-91a1-4fa3-a007-4d6f3e3b470b/communities/images/lookup.png) to search for the file.
9. Save your work.

#### See Also

- [Best Practices and Tips for Using SEO in Experience Builder Sites](https://help.salesforce.com/s/articleView?id=experience.networks_seo_best_practices.htm&release=264.0.0&type=5 "Follow these guidelines and best practices when optimizing SEO in your site.")
- [SEO for Experience Builder Sites FAQ](https://help.salesforce.com/s/articleView?id=experience.networks_seo_faq.htm&release=264.0.0&type=5 "Answers to common questions about using SEO in your Experience Builder site.")