---
schema_version: "1"
id: 1ee99c6bcb972421bc09f86a98f5a490c4ba389a8352d2ae4c9214e20faa69f8
title: Set Up a Web Content Connection (Sitemap)
collection: developer
version: current
locale: en-us
portal_url: https://docs.salesforce.com/developer/current/en-us/data-cloud/data-cloud-int/guides/webcontentsitemap/c360-a-set-up-a-web-content-connection-sitemap.md
canonical_url: https://developer.salesforce.com/docs/data/data-cloud-int/guide/c360-a-set-up-a-web-content-connection-sitemap.html
source_site: developer.salesforce.com
content_type: documentation
content_hash: 2d1d53f24505519ad1101ec75edaeadc4fdf70afa540f35105272c587b626602
products:
  - Data 360 Integration Guide
  - Data 360 Developer Documentation
---
# Set Up a Web Content Connection (Sitemap)

Set up the Web Content connection (Sitemap) to start the flow of data into Data 360.

| User Permissions Needed |                                                           |
| ----------------------- | --------------------------------------------------------- |
| To create a connection: | System Admin profile or Data 360 Architect permission set |

**note:** It’s your obligation to ensure that you have the rights to the data collected using this feature. Salesforce disclaims all liability with respect to such data collected.

Before you begin:

- Verify that your admin has enabled firewalls on the system you want Data 360 to connect to by including these [IP addresses](https://help.salesforce.com/s/articleView?id=data.c360_a_data_cloud_ip_address_allowlist.htm&type=5) to your allowlists.

1.  In Data 360, click **Setup**, and select **Data 360 Setup**.

2.  Under **External Integrations**, select **Other Connectors**.

3.  Click **New**.

4.  On the Source tab, select **Web Content (Sitemap)** and click **Next**.

5.  Enter a connection name and a connection API name.

6.  To allow the sitemap connector to access your website, select an authentication method from the dropdown menu. If your website requires credentials, enter a username and password. Make sure that the username you provide has the necessary access to the content you want to ingest. The connector gets the same access permissions as this user.

7.  Provide a valid URL of an XML sitemap ending with `/sitemap.xml` (for example: `https://help.yourdomain.com/sitemap.xml`). URLs for Gzip-compressed XML sitemaps are also supported.

8.  If your website requires a specific user agent, customize the **User Agent** field string. Otherwise, the connector uses the default user agent.

9.  To enhance content ingestion, override any of the site’s robots.txt file directives. You can ignore the website’s robots.txt file only if you have rights to its contents and IT approval.

    - To prevent ingestion issues caused by your website’s robots.txt file, select the checkbox next to **Ignore robots.txt entirely**.
    - To prevent ingestion failures caused by crawl delay exceeding 1 second, select the checkbox next to **Ignore only Crawl-Delay in robots.txt**.
    - To prevent site page exclusion from ingestion, select the checkbox next to **Ignore only Disallow in robots.txt**.

10. To prevent ingestion timeouts on large sites, increase the value for **Requests per Second** (crawling rate). On small websites, prevent ingestion blocking by disabling or configuring a lower value for **Requests per Second**. By default, the connector performs five requests per second.

11. To review your configuration, click **Test Connection**.

12. Click **Save**.

At this point, Data 360 creates the connection, and you can now create data streams.

## Next Step

- [Create a Web Content Sitemap UDLO](https://developer.salesforce.com/docs/data/data-cloud-int/guide/c360-a-create-sitemap-udlo.md)