---
schema_version: "1"
id: 033c7b94a7c0b0612a9e484f234e575f8f50978b7a4d04d85b8e460a4dbd3d80
title: Set Up a Web Content Connection (Crawler)
collection: developer
version: current
locale: en-us
portal_url: https://docs.salesforce.com/developer/current/en-us/data-cloud/data-cloud-int/guides/webcontentcrawler/c360-a-set-up-a-web-content-connection-crawler.md
canonical_url: https://developer.salesforce.com/docs/data/data-cloud-int/guide/c360-a-set-up-a-web-content-connection-crawler.html
source_site: developer.salesforce.com
content_type: documentation
content_hash: bfb0ce94c6b5824fa08593afafac620c9892cc283e344fbf30a0e558ee2722a4
products:
  - Data 360 Integration Guide
  - Data 360 Developer Documentation
---
# Set Up a Web Content Connection (Crawler)

Set up the Web Content connection (Crawler) to start the flow of data into Data 360.

| User Permissions Needed |                                                           |
| ----------------------- | --------------------------------------------------------- |
| To create a connection: | System Admin profile or Data 360 Architect permission set |

**note:** It’s your obligation to ensure that you have the rights to the data collected using this feature. Salesforce disclaims all liability with respect to such data collected.

Before you begin:

- Verify that your admin has enabled firewalls on the system you want Data 360 to connect to by including these [IP addresses](https://help.salesforce.com/s/articleView?id=data.c360_a_data_cloud_ip_address_allowlist.htm&type=5) to your allowlists.

1.  In Data 360, click **Setup**, and select **Data 360 Setup**.

2.  Under **External Integrations**, select **Other Connectors**.

3.  Click **New**.

4.  On the Source tab, select **Web Content (Crawler)** and click **Next**.

5.  Enter a connection name and a connection API name.

6.  To allow the crawler to access your website, select an authentication method from the dropdown menu. If your website requires credentials, enter a username and password. Make sure that the username you provide has the necessary access to the content you want to ingest. The connector gets the same access permissions as this user.

7.  To set a starting point for the website ingestion, provide a starting URL, for example: `https://acme.com`. Don’t include any spaces in this field.

8.  If your website requires a specific user agent, customize the **User Agent** field string. Otherwise, the connector uses the default user agent.

9.  To enhance content ingestion, override any of the site’s robots.txt file directives. You can ignore the website’s robots.txt file only if you have rights to its contents and IT approval.

    - To prevent ingestion issues caused by your website’s robots.txt file, select the checkbox next to **Ignore robots.txt entirely**.
    - To prevent ingestion failures caused by crawl delay exceeding 1 second, select the checkbox next to **Ignore only Crawl-Delay in robots.txt**.
    - To prevent site page exclusion from ingestion, select the checkbox next to **Ignore only Disallow in robots.txt**.

10. To prevent ingestion timeouts on large sites, increase the value for **Requests per Second** (crawling rate). On small websites, prevent ingestion blocking by disabling or configuring a lower value for **Requests per Second**. By default, the connector performs five requests per second.

11. To review your configuration, click **Test Connection**.

12. Click **Save**.

At this point, Data 360 creates the connection, and you can now create data streams.

## Next Step

- [Create a Web Content Crawler UDLO](https://developer.salesforce.com/docs/data/data-cloud-int/guide/c360-a-create-crawler-udlo.md)