--- schema_version: "1" id: fefab6ec8d29dffbd3a59608fb49ecfe0a26306a9be80ab0898828c2d686aed2 title: Configuring Crawl Sources collection: mulesoft version: current locale: en-us portal_url: https://docs.salesforce.com/mulesoft/current/en-us/mulesoft-webcrawler-connector/latest/configuring-crawl-sources.md canonical_url: https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/configuring-crawl-sources source_site: docs.mulesoft.com content_type: documentation content_hash: 6fab48e9adfcfa4949c8e82bb9cb5e77a2e10ef66b4b80f2eb8795b7e0671dfd description: MuleSoft Documentation Site release: "1.0" latest: true --- --- product: MuleSoft WebCrawler Connector version: 1.0 is-latest-version: true --- # Configuring Crawl Sources > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Configure the **On Content Crawl** source. ## Configure the On Content Crawl Source The **On Content Crawl** source crawls a website starting from a configured path, emitting each discovered page, image, or document as a Mule event. The source uses a VM queue for URL scheduling and an object store for URL deduplication, enabling distributed and resumable crawling. To configure the **On Content Crawl** source: - **Path** Relative path, resolved against the connection’s base URL, where the crawl starts. When empty or `/`, the crawl starts at the base URL itself. Fully-qualified URLs are rejected with INVALID\_PARAMETERS. - **Additional seed URLs** Comma-separated list of additional fully-qualified seed URLs to enqueue alongside the path. Each entry must be a fully-qualified URL, and relative paths are dropped. Off-host and off-path entries are dropped and a warning is logged. - **Maximum depth** The maximum depth to reach while crawling a website. Defaults to 0. - **Queue Name** VM queue used to schedule URLs for crawling. When omitted, defaults to `webcrawler-{configName}-source-queue`. - **Object Store** Object store used for crawl visit fingerprints (content hash, ETag, last-modified, and correlation ID) and the active crawl identity. Use a persistent store so that reconnection and app restart resume the same crawl. - **Output format** Format applied to each emitted page body. Select TEXT, HTML, or MARKDOWN. Default is TEXT. - **Number Of Consumers** Number of concurrent queue consumers used by this source. Defaults to 1. - **Hash route mode** How to treat URLs containing #fragments. IGNORE strips fragments before crawling (default, fits classic server-rendered sites). PATH rewrites SPA-style hash routes (for example, #/page) into real paths (/page). PRESERVE keeps the URL verbatim, fragment included. - **Crawl schedule** Periodic re-crawl mode. Select **No Schedule** to crawl once at start (default), or **Interval** to re-seed the crawl at a fixed frequency and time unit. Ticks that occur while a crawl is in progress are coalesced into one pending reseed, so consumers don’t stack. The schedule is registered on all replicas, but the actual reseed runs on the primary node only. - **Encoding** Encoding for emitted page bodies. Overrides the configuration’s encoding when set. - **Restrict crawl under URL** If true, only internal pages under the base URL are crawled. Defaults to false. - **Regex URLs filter logic** Type of filter logic to apply to regex URLs while crawling. - **Regex URLs List** List of regex patterns for URLs to include or exclude while crawling. ### Target Content Options By default, the source emits only page content. To also download and emit the images and documents referenced on each crawled page, enable the **Images** and **Documents** options. Use the **Max image number for each page** and **Max document number for each page** options to cap how many assets are fetched per page, and **Additional Allowed Base URLs** to control which hosts assets can be downloaded from. - **Pages** If true, page content of the crawled URL is emitted. Defaults to true. - **Images** If true, images referenced on the crawled URL are emitted. Defaults to false. - **Max image number for each page** Maximum number of images to fetch for each page. 0 means no limit. - **Documents** If true, documents referenced on the crawled URL are emitted. Defaults to false. - **Max document number for each page** Maximum number of documents to fetch for each page. 0 means no limit. - **Additional Allowed Base URLs** Optional comma-separated allowlist for image and document URLs. When set, assets must be under the base URL or one of these base URLs. - **Retrieve meta tags** If true, meta tags are retrieved from each crawled page. Defaults to false. - **Tag list** List of HTML tags for which content must be retrieved. ### Crawl Options - **Delay between pages** The delay introduced between each request, in the time unit selected below. Defaults to 0. - **Delay time unit** Time unit for the delay between pages. - **Enforce robots.txt** If true, enforce checks against the robots.txt file. Defaults to false. ### Page Load Options - **Wait on page load** The time to wait on page load, in the time unit selected below. On the HTTP connection, values greater than 0 fail with INVALID\_PARAMETERS. - **Wait on page load unit** Time unit for the page-load wait. ### WebDriver Options (Remote WebDriver only) These options apply only when using the Remote WebDriver connection type: - **Wait for XPath** The XPath to wait for before proceeding with content extraction. - **Extract Shadow DOM** If true, extracts the Shadow DOM content. - **Shadow Host(s) XPath** Shadow host(s) to extract by XPath. If not set, the whole Shadow DOM is extracted. ### Timeout Options - **Response Timeout** Response timeout value. Overrides the configuration’s timeout when set. - **Response Timeout Unit** Time unit for the response timeout. ### XML Configuration This is the XML for this source: ```xml ``` ### Output Configuration Each emitted event contains the crawled page content in the configured output format. The response attributes include: - `cacheControl` The cache-control header value from the page response. - `contentType` The content type of the page response. - `depth` The crawl depth at which this page was discovered. - `etag` The ETag header value from the page response. - `eventType` The type of emitted event: PAGE, IMAGE, or DOCUMENT. - `fileName` The file name of the emitted image or document. - `lastModified` The last-modified timestamp from the page response. - `metaTags` The meta tags retrieved from the page. - `size` The size of the emitted content in bytes. - `statusCode` The HTTP status code of the page response. - `title` The page title. - `url` The page URL. - `otherAttributes` Additional metadata about the page. ## See Also - [MuleSoft WebCrawler Connector Overview](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/) - [Configuring Crawl Operations](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/configuring-crawl-operations) - [MuleSoft WebCrawler Connector Reference](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-reference) - [Salesforce Help](https://help.salesforce.com)