---
schema_version: "1"
id: 2f51e66208f572c6c7a3cdc0c22dfefc8d81bb6c46c9ff774a6da51e9b700868
title: Using Anypoint Studio to Configure MuleSoft WebCrawler Connector 1.0
collection: mulesoft
version: current
locale: en-us
portal_url: https://docs.salesforce.com/mulesoft/current/en-us/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-studio.md
canonical_url: https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-studio
source_site: docs.mulesoft.com
content_type: documentation
content_hash: aff0233534ef51e6f99e254a2f103fa978dc36784dcab8e4fc20d57979ccc99e
description: MuleSoft Documentation Site
release: "1.0"
latest: true
---
---
product: MuleSoft WebCrawler Connector
version: 1.0
is-latest-version: true
---

# Using Anypoint Studio to Configure MuleSoft WebCrawler Connector 1.0

> For the full documentation index, see: https://docs.mulesoft.com/llms.txt

Anypoint Studio (Studio) editors help you design and update your Mule applications, properties, and
configuration files.

To add and configure a connector in Studio:

1.  [Create a Mule project](#create-mule-project).

2.  [Add the connector to your Mule project](#add-connector-to-project).

3.  [Configure a source for the connector’s flow](#configure-source).

4.  [Add a connector operation to the flow](#add-connector-operation).

5.  [Configure a global element for the connector](#configure-global-element).

6.  [Configure the other connector fields](#configure-other-fields).

When you run the connector, you can view the app log to check for problems in real time, as
described in [View the App Log](#view-app-log).

If you are new to configuring connectors in Studio, see
[Using Anypoint Studio to Configure a Connector](https://docs.mulesoft.com/connectors/introduction/intro-config-use-studio).
If, after reading this topic, you need additional information about the connector fields, see the
[MuleSoft WebCrawler Connector Reference](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-reference).

## Create a Mule Project

In Studio, create a new Mule project in which to add and configure Anypoint Connector for MuleSoft
WebCrawler (MuleSoft WebCrawler Connector):

1.  In Studio, select **File > New > Mule Project**.

2.  Enter a name for your Mule project and click **Finish**.

## Add the Connector to Your Mule Project

Add MuleSoft WebCrawler Connector to your Mule project to automatically populate the XML code with
the connector’s namespace and schema location and add the required dependencies to the project’s
`pom.xml` file:

1.  In **Mule Palette**, click **(X) Search in Exchange**.

2.  In **Add Dependencies to Project**, type `mulesoft webcrawler` in the search field.

3.  Click **MuleSoft WebCrawler Connector** in **Available modules**.

4.  Click **Add**.

5.  Click **Finish**.

Adding a connector to a Mule project in Studio doesn’t make that connector available to other
projects in your Studio workspace.

## Configure a Source

A source initiates a flow when a specified condition is met. You can configure one of these sources
to use with MuleSoft WebCrawler Connector:

- **On Content Crawl**

  Crawls a website starting from a configured path, emitting each discovered page, image, or
  document as a Mule event. Supports deduplication via an object store and URL scheduling via a VM
  queue.

- **HTTP Listener**

  Initiates a flow each time it receives a request on the configured host and port.

- **Scheduler**

  Initiates a flow when a time-based condition is met.

For example, to configure an **HTTP > Listener** source, follow these steps:

1.  In **Mule Palette**, select **HTTP > Listener**.

2.  Drag **Listener** to the Studio canvas.

3.  On the **Listener** configuration screen, optionally change the value of the **Display Name**
    field.

4.  Specify a value for the **Path** field.

5.  Click the plus sign (**+**) next to the **Connector configuration** field to configure a global
    element that can be used by all instances of the **HTTP > Listener** source in the app.

6.  On the **General** tab, specify the connection information for the connector.

7.  On the **TLS** tab, optionally specify the TLS information for the connector.

8.  On the **Advanced** tab, optionally specify reconnection information, including a reconnection
    strategy.

9.  Click **Test Connection** to confirm that Mule can connect with the specified server.

10. Click **OK**.

## Add a Connector Operation to the Flow

When you add a connector operation to your flow, you are specifying an action for that connector to
perform.

To add an operation for MuleSoft WebCrawler Connector:

1.  In **Mule Palette**, select **MuleSoft WebCrawler Connector** and then select the operation to
    add.

2.  Drag the operation onto the Studio canvas, next to the source.

## Configure a Global Element for the Connector

When you configure a connector, configure a global element that all instances of that connector in
the app can use. Configuring a global element requires you to provide the connection settings that
the connector requires to access the target web resources.

To configure the global element for MuleSoft WebCrawler Connector:

1.  Select the operation in the Studio canvas.

2.  In the **General** configuration screen for the operation, click the **Add** icon to access the
    global element configuration fields.

3.  In the **General** tab, in **Connection**, select the connection type to configure:

    - [HTTP](#http)

    - [Remote WebDriver](#remote-webdriver)

    You can reference a configuration file that contains ANT-style property placeholders
    (recommended), or you can enter your connection settings in the global configuration properties.
    For information about the benefits of using property placeholders and how to configure them,
    refer to
    [Anypoint Connector Configuration](https://docs.mulesoft.com/connectors/introduction/intro-connector-configuration-overview).

4.  In the **Advanced** tab, optionally specify reconnection information, including a reconnection
    strategy.

5.  Click **Test Connection** to confirm that Mule can connect with the specified server.

6.  Click **OK**.

### HTTP

Use the HTTP connection type for crawling static web content.

Enter this information in the **General** tab of the **Global Element Properties** screen to
configure the HTTP connection:

<table><colgroup><col> <col></colgroup><thead><tr><th>Field</th><th>User Action</th></tr></thead><tbody><tr><td><p><strong>Name</strong></p></td><td><div><div><p>Enter the configuration name.</p></div></div></td></tr><tr><td><p><strong>Connection</strong></p></td><td><div><div><p>Select <strong>HTTP</strong>.</p></div></div></td></tr><tr><td><p><strong>Base URL</strong></p></td><td><div><div><p>Enter the base URL against which relative paths are resolved.</p></div></div></td></tr><tr><td><p><strong>Browser User Agent</strong></p></td><td><div><div><p>Enter the browser user-agent advertised on outbound requests.</p></div></div></td></tr><tr><td><p><strong>Proxy Configuration</strong></p></td><td><div><div><p>Configure a proxy for outbound HTTP requests. Specify host, port, and optional credentials.</p></div></div></td></tr><tr><td><p><strong>Response timeout</strong></p></td><td><div><div><p>Enter the connect and read timeout for the connection’s HTTP client. Default is 30 seconds.</p></div></div></td></tr><tr><td><p><strong>Response timeout unit</strong></p></td><td><div><div><p>Select the time unit for the response timeout.</p></div></div></td></tr></tbody></table>

Optionally configure TLS settings for secure connections in the **TLS** tab.

### Remote WebDriver

Use the Remote WebDriver connection type for crawling dynamic web content that requires JavaScript
rendering.

Enter this information in the **General** tab of the **Global Element Properties** screen to
configure the Remote WebDriver connection:

<table><colgroup><col> <col></colgroup><thead><tr><th>Field</th><th>User Action</th></tr></thead><tbody><tr><td><p><strong>Name</strong></p></td><td><div><div><p>Enter the configuration name.</p></div></div></td></tr><tr><td><p><strong>Connection</strong></p></td><td><div><div><p>Select <strong>Remote WebDriver</strong>.</p></div></div></td></tr><tr><td><p><strong>Base URL</strong></p></td><td><div><div><p>Enter the base URL against which relative paths are resolved.</p></div></div></td></tr><tr><td><p><strong>Remote Browser URL</strong></p></td><td><div><div><p>Enter the URL of your own remote Selenium WebDriver endpoint (for example, Selenium Grid, standalone ChromeDriver, or BrowserStack). Chromium-only (Chrome, Chromium, Edge).</p></div></div></td></tr><tr><td><p><strong>Browser User Agent</strong></p></td><td><div><div><p>Enter the browser user-agent advertised on outbound requests.</p></div></div></td></tr><tr><td><p><strong>Browser Authentication</strong></p></td><td><div><div><p>Select the authentication used to reach the remote browser endpoint: None, Basic, or API Access Token.</p></div></div></td></tr><tr><td><p><strong>Proxy Configuration</strong></p></td><td><div><div><p>Configure a proxy for outbound HTTP requests. Specify host, port, and optional credentials.</p></div></div></td></tr><tr><td><p><strong>Response timeout</strong></p></td><td><div><div><p>Enter the connect and read timeout for the connection’s HTTP client. Default is 30 seconds.</p></div></div></td></tr><tr><td><p><strong>Response timeout unit</strong></p></td><td><div><div><p>Select the time unit for the response timeout.</p></div></div></td></tr></tbody></table>

Optionally configure TLS settings for secure connections in the **TLS** tab. You can also configure
a **Pooling Profile** in the **Advanced** tab to control connection pool behavior.

## Configure Additional Connector Fields

After you configure a global element for the connector, configure the other required fields. The
required fields vary depending on which connector operation you use.

For detailed configuration information for each operation type, see
[MuleSoft WebCrawler Connector - Additional Configuration Information](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-additional-config).

## View the App Log

To check for problems, you can view the app log as follows:

- If you’re running the app from Anypoint Platform, the app log output is visible on the Anypoint
  Studio console window.

- If you’re running the app using Mule from the command line, the app log output is visible in your
  operating system console.

Unless the log file path is customized in the app’s log file (`log4j2.xml`), you can also view the
app log in the default location `MULE_HOME/logs/<app-name>.log`. You can configure the location of
the log path in the app log file `log4j2.xml`.

## See Also

- [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors)

- [Using Studio to Configure a Connector](https://docs.mulesoft.com/connectors/introduction/intro-config-use-studio)

- [MuleSoft WebCrawler Connector Reference](https://docs.mulesoft.com/mulesoft-webcrawler-connector/latest/mulesoft-webcrawler-connector-reference)

- [Salesforce Help](https://help.salesforce.com)