---
schema_version: "1"
id: 59952fc250b2bb0c40b6b5c588837b84dec248fa68f53a1d0179ac062822fa63
title: Configuring Crawl Operations
collection: mulesoft
version: current
locale: en-us
portal_url: https://docs.salesforce.com/mulesoft/current/en-us/mulesoft-webcrawler-connector/latest/configuring-crawl-operations.md
canonical_url: https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/configuring-crawl-operations
source_site: docs.mulesoft.com
content_type: documentation
content_hash: dc116d1a01c28a8e9865396fd0a47e2cf1a6603d12efba3eab8d8e0877ceaf4c
description: MuleSoft Documentation Site
release: "1.0"
latest: true
---
---
product: MuleSoft WebCrawler Connector
version: 1.0
is-latest-version: true
---

# Configuring Crawl Operations

> For the full documentation index, see: https://docs.mulesoft.com/llms.txt

Configure the **Get SiteMap** operation.

## Configure the Get SiteMap Operation

The **Get SiteMap** operation returns a sitemap in standard XML format following the
[sitemap protocol](https://www.sitemaps.org/protocol.html). The operation first tries to discover
URLs from the site’s own sitemap and robots.txt, then falls back to a breadth-first crawl up to the
specified depth. The sitemap results can be used to customize subsequent crawling of specific pages.

Priority is calculated based on depth: depth 0 = priority 1.0, depth 1 = priority 0.9, and so on.
Depth greater than 10 always yields priority 0.0.

To configure the **Get SiteMap** operation:

1.  Select the operation on the Anypoint Code Builder or Studio canvas.

2.  In the **General** properties tab for the operation, enter these values:

    - **Path**

      Relative path, resolved against the connection’s base URL, where the sitemap crawl starts.

    - **Maximum depth**

      The maximum depth to reach while crawling a website. Defaults to 0.

    - **Encoding**

      Encoding for emitted page bodies. Overrides the configuration’s encoding when set.

    - **Restrict crawl under URL**

      If true, only internal pages under the base URL are crawled. Defaults to false.

    - **Regex URLs filter logic**

      Type of filter logic to apply to regex URLs while crawling. Select INCLUDE to crawl only
      matching pages, or EXCLUDE to skip matching pages.

    - **Regex URLs List**

      List of regex patterns for URLs to include or exclude while crawling, based on the filter
      logic selected above.

### Crawl Options

- **Delay between pages**

  The delay introduced between each request, in the time unit selected below. Defaults to 0.

- **Delay time unit**

  Time unit for the delay between pages.

- **Enforce robots.txt**

  If true, enforce checks against the robots.txt file. Defaults to false.

### Page Load Options

- **Wait on page load**

  The time to wait on page load, in the time unit selected below. On the HTTP connection, values
  greater than 0 fail with INVALID\_PARAMETERS.

- **Wait on page load unit**

  Time unit for the page-load wait.

### WebDriver Options (Remote WebDriver only)

These options apply only when using the Remote WebDriver connection type:

- **Wait for XPath**

  The XPath to wait for before proceeding with content extraction.

- **Extract Shadow DOM**

  If true, extracts the Shadow DOM content.

- **Shadow Host(s) XPath**

  Shadow host(s) to extract by XPath. If not set, the whole Shadow DOM is extracted.

### Timeout Options

- **Response Timeout**

  Response timeout value. Overrides the configuration’s timeout when set.

- **Response Timeout Unit**

  Time unit for the response timeout.

### XML Configuration

This is the XML for this operation:

```xml
<ms-webcrawler:get-sitemap
    config-ref="MuleSoft_WebCrawler_Config"
    doc:name="Get SiteMap"
    doc:id="6410aae9-21d4-4005-8d86-ac9a136657b4"
    path="#[payload.path]"
    maxDepth="#[payload.maxDepth]"
    restrictToPath="true"/>
```

### Output Configuration

This operation responds with an XML payload that uses the sitemap protocol specification. This is an
example response:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2024-01-15T10:30:00Z</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/docs</loc>
    <priority>0.9</priority>
  </url>
  <url>
    <loc>https://example.com/docs/getting-started</loc>
    <priority>0.8</priority>
  </url>
</urlset>
```

The `<lastmod>` and `<changefreq>` elements appear only for URLs where the crawled site’s own source
sitemap supplies them, so some `<url>` entries include them while others contain only `<loc>` and
`<priority>`.

The response attributes include:

- `count`

  The total number of URLs in the sitemap.

- `depth`

  The configured maximum crawl depth.

- `url`

  The root URL of the crawled site.

## See Also

- [MuleSoft WebCrawler Connector Overview](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/)

- [Configuring Crawl Sources](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/configuring-crawl-sources)

- [MuleSoft WebCrawler Connector Reference](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-reference)

- [Salesforce Help](https://help.salesforce.com)