--- schema_version: "1" id: 25b33ca097f6cf6c9773e041d1853e732225933f98342e3bdfaae2de8ca724c9 title: MuleSoft WebCrawler Connector 1.0 collection: mulesoft version: current locale: en-us portal_url: https://docs.salesforce.com/mulesoft/current/en-us/mulesoft-webcrawler-connector/latest/index.md canonical_url: https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/ source_site: docs.mulesoft.com content_type: documentation content_hash: 36468049d460f4dbe79cacd319e6ae772f400bb1d8ed9612a5c72815604c407b description: MuleSoft Documentation Site release: "1.0" latest: true --- --- product: MuleSoft WebCrawler Connector version: 1.0 is-latest-version: true --- # MuleSoft WebCrawler Connector 1.0 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for MuleSoft WebCrawler (MuleSoft WebCrawler Connector) is a highly customizable web-crawling connector that enables you to crawl websites, retrieve content, and automate web scraping processes within Mule flows. The connector is suited for gathering large-scale web data efficiently, with helper operations for fine-tuned crawling such as targeting specific content types and emitting referenced documents and images as Mule events. Use it to organize web data into structured formats for knowledge extraction, AI applications, and downstream processing. For information about compatibility and fixed issues, see the MuleSoft WebCrawler Connector release notes. ## Before You Begin To use this connector, you must be familiar with: - Anypoint Connectors - Mule runtime engine (Mule) - Elements and global elements in a Mule flow - How to create a Mule app using Anypoint Code Builder or Anypoint Studio - Web crawling concepts and robots.txt compliance Before creating an app, you must have: - Java 17 (required for compilation and runtime) - Apache Maven - Anypoint Platform - The latest versions of Anypoint Code Builder or Anypoint Studio - Your own Chromium-based remote WebDriver endpoint when content must be rendered in a real browser before scraping, such as JavaScript-heavy web applications. MuleSoft doesn’t provide or host a remote browser, so you must supply and maintain your own. ## Connection Types MuleSoft WebCrawler Connector supports these connection types for different crawling scenarios. - **HTTP** Connects via standard HTTP requests with a configurable base URL, browser user agent, proxy, response timeout, and TLS settings. Use for static web content that doesn’t require JavaScript rendering. - **Remote WebDriver** Connects through a remote Selenium WebDriver endpoint that you provide. Use for JavaScript-heavy or single-page application (SPA) pages that require browser rendering. ## Key Features MuleSoft WebCrawler Connector simplifies web data extraction into Mule applications with: - Comprehensive Web Crawling Crawl entire websites or target specific content for large-scale data extraction with configurable depth and URL filtering. - Customizable Crawls Control crawl depth, content filters, URL inclusion or exclusion patterns, and crawl delays tailored to specific business needs. - Content Extraction Retrieve page content in TEXT, HTML, or MARKDOWN format. Target specific HTML tags to extract only relevant content. - Document and Image Extraction Emit documents and images referenced on crawled pages as Mule events, with configurable limits and an optional allowlist of base URLs. - Page Elements Get analytical data about web pages including word count, element or tag counts, and link structures to help customize crawls. - Dynamic Content Support Crawl JavaScript-rendered pages using the Remote WebDriver connection, including Shadow DOM extraction and XPath-based wait conditions. - Robots.txt Compliance Optionally enforce robots.txt rules to ensure respectful crawling practices. - Event-Driven Crawling Use the **On Content Crawl** source to emit each discovered page, image, or document as a Mule event, with deduplication and URL queue management. ## Use Cases MuleSoft WebCrawler Connector supports a wide range of data ingestion and automation scenarios.
RoleGoal

Agentic Automation

Scrape internal documents to ground agents and bots with up-to-date knowledge.

Data Analyst

Extract product information from e-commerce websites for competitive analysis.

Marketing Team

Gather customer reviews from various platforms for sentiment tracking.

Business Development

Collect contact information from industry directories for lead generation.

## Authentication MuleSoft WebCrawler Connector supports unauthenticated (public) websites. ## Limitations MuleSoft WebCrawler Connector has these limitations: - Analyzing or extracting the content within multimodal media, such as images and videos, isn’t supported. The connector can emit referenced image and document files as Mule events, but it doesn’t interpret what those files contain. - The recommended maximum size for a downloadable attachment is 25 MB. ## Performance The total crawl time depends on several factors, including: - Crawl depth Higher crawl depths discover and process more pages. - Number of pages Larger websites naturally take longer to crawl. - Website complexity JavaScript-rendered websites require a Remote Browser and generally take longer than static HTML sites. - Network latency Response times between the crawler, Remote Browser, and target website can affect overall performance. - Attachments Downloading large files, such as PDFs or images, increases crawl time. - Remote Browser capacity The CPU, memory, and concurrency of the Remote Browser deployment directly affect crawling throughput. - Number of consumers Crawling throughput depends on the number of concurrent queue consumers. More consumers can increase throughput, but the Remote Browser must be able to handle that many concurrent sessions. ## Best Practices For the best crawling experience: - Deploy the Remote Browser with sufficient CPU and memory based on your expected crawl size. - Configure the Remote Browser reconnection strategy to Reconnect Forever to improve resilience during long-running crawls. - Use the HTTP connection for static websites whenever possible, and reserve the Remote Browser for JavaScript-rendered or interactive websites. - Limit crawl depth and exclude unnecessary pages or attachments to reduce crawl time. - For large websites, break the crawl into smaller jobs or use sitemap-based discovery to target only relevant pages. - Keep the crawl schedule in sync with the object store’s time-to-live (TTL) so that a full crawl runs on reseed. Otherwise, reseeding is skipped for that tick. ## Next Step After you complete the prerequisites, you are ready to create an app and configure the connector using [Anypoint Studio](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-studio) or [Anypoint Code Builder](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-config-acb). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [MuleSoft WebCrawler Connector Reference](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-reference) - [Salesforce Help](https://help.salesforce.com)