--- schema_version: "1" id: 2af7e444ab23616b6b7572d676ae30413b0ad5a9b3f5c2341c63386ec9c130eb title: Configuring Page Operations collection: mulesoft version: current locale: en-us portal_url: https://docs.salesforce.com/mulesoft/current/en-us/mulesoft-webcrawler-connector/latest/configuring-page-operations.md canonical_url: https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/configuring-page-operations source_site: docs.mulesoft.com content_type: documentation content_hash: 970f950d88b7707292abe587dd2b94015cc96b09d45eea0a6619a2a09900a1ab description: MuleSoft Documentation Site release: "1.0" latest: true --- --- product: MuleSoft WebCrawler Connector version: 1.0 is-latest-version: true --- # Configuring Page Operations > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Configure the **Get Page Content** and **Get Page Elements** operations. ## Configure the Get Page Content Operation The **Get Page Content** operation retrieves the content of a specified web page in TEXT, HTML, or MARKDOWN format. Optionally target specific HTML tags via the Tag list parameter. To configure the **Get Page Content** operation: 1. Select the operation on the Anypoint Code Builder or Studio canvas. 2. In the **General** properties tab for the operation, enter these values: - **Path** Relative path, resolved against the connection’s base URL, of the page to retrieve content from. - **Output format** Format of the extracted content. Select TEXT, HTML, or MARKDOWN. Default is TEXT. - **Encoding** Encoding for emitted page bodies. Overrides the configuration’s encoding when set. - **Tag list** List of HTML tags for which content must be retrieved. If specified, only content within the listed tags is returned. - **Javascript** Browser-context JavaScript executed after page load (not available for the HTTP connection). Trusted input only — never bind to user-controlled data; runs with full DOM access. ### Crawl Options - **Enforce robots.txt** If true, enforce checks against the robots.txt file. Defaults to false. ### Page Load Options - **Wait on page load** The time to wait on page load, in the time unit selected below. On the HTTP connection, values greater than 0 fail with INVALID\_PARAMETERS. - **Wait on page load unit** Time unit for the page-load wait. ### WebDriver Options (Remote WebDriver only) These options apply only when using the Remote WebDriver connection type: - **Wait for XPath** The XPath to wait for before proceeding with content extraction. - **Extract Shadow DOM** If true, extracts the Shadow DOM content. - **Shadow Host(s) XPath** Shadow host(s) to extract by XPath. If not set, the whole Shadow DOM is extracted. ### Timeout Options - **Response Timeout** Response timeout value. Overrides the configuration’s timeout when set. - **Response Timeout Unit** Time unit for the response timeout. ### XML Configuration This is the XML for this operation: ```xml ``` ### Output Configuration This operation responds with a JSON payload. This is an example response: ```json { "title": "Example Page Title", "url": "https://example.com/page", "content": "The full text content of the page..." } ``` - `title` The page title. - `url` The page URL. - `content` The extracted content in the specified output format. ## Configure the Get Page Elements Operation The **Get Page Elements** operation fetches analytical data about a web page including word count, element or tag counts (H1–H5, DIV, P), link structures (internal, external, reference), and image links. Use these metrics to customize subsequent crawling operations. To configure the **Get Page Elements** operation: 1. Select the operation on the Anypoint Code Builder or Studio canvas. 2. In the **General** properties tab for the operation, enter these values: - **Path** Relative path, resolved against the connection’s base URL, of the page to analyze. - **Encoding** Encoding for emitted page bodies. Overrides the configuration’s encoding when set. - **Tag list** List of HTML tags for which content must be retrieved. - **Javascript** Browser-context JavaScript executed after page load (not available for the HTTP connection). Trusted input only — never bind to user-controlled data; runs with full DOM access. ### Crawl Options - **Enforce robots.txt** If true, enforce checks against the robots.txt file. Defaults to false. ### Page Load Options - **Wait on page load** The time to wait on page load, in the time unit selected below. On the HTTP connection, values greater than 0 fail with INVALID\_PARAMETERS. - **Wait on page load unit** Time unit for the page-load wait. ### WebDriver Options (Remote WebDriver only) These options apply only when using the Remote WebDriver connection type: - **Wait for XPath** The XPath to wait for before proceeding with content extraction. - **Extract Shadow DOM** If true, extracts the Shadow DOM content. - **Shadow Host(s) XPath** Shadow host(s) to extract by XPath. If not set, the whole Shadow DOM is extracted. ### Timeout Options - **Response Timeout** Response timeout value. Overrides the configuration’s timeout when set. - **Response Timeout Unit** Time unit for the response timeout. ### XML Configuration This is the XML for this operation: ```xml ``` ### Output Configuration This operation outputs a Binary payload containing page statistics and link details in JSON format. This is an example response: ```json { "pageStats": { "div": 36, "p": 6, "reference": 0, "internal": 7, "external": 2, "images": 4, "wordCount": 147, "h1": 1, "h2": 0, "h3": 4, "h4": 0, "h5": 0 }, "links": { "reference": [], "internal": ["https://example.com/docs", "https://example.com/about"], "external": ["https://github.com/example"], "images": ["https://example.com/images/logo.png"], "documents": [], "iframe": [] }, "title": "Example Page", "url": "https://example.com/" } ``` - `pageStats` - Counts of HTML elements and page metrics. - `div`, `p`, `h1` through `h5` - Counts of the respective HTML elements. - `reference`, `internal`, `external` - Counts of reference, internal, and external links. - `images` - Count of images on the page. - `wordCount` - Total word count of the page content. - `links` - Categorized lists of URLs found on the page. - `reference` - Reference links (anchors). - `internal` - Internal links to the same domain. - `external` - External links to other domains. - `images` - Image URLs. - `documents` - Document URLs. - `iframe` - Iframe URLs. - `title` - The page title. - `url` - The page URL. ## See Also - [MuleSoft WebCrawler Connector Overview](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/) - [MuleSoft WebCrawler Connector Reference](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-reference) - [Salesforce Help](https://help.salesforce.com)