# mulesoft Apache Kafka Connector - Full Documentation --- product: Apache Kafka Connector version: 4.10 --- # Apache Kafka 4.10 Examples > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following examples shows how to configure Apache Kafka Connector: - [Publish and Consume Example](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-publish-consume-example) Use the **Publish** and **Consume** to publish a message to Apache Kafka and then retrieve it. - [Integrate Confluent Schema Registry](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-configure-confluent-example) Configure Confluent Schema Registry via the **Additional Properties** field using Kafka Connector. - [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-bulk-publish-example) Use **Bulk Publish** to publish multiple messages to Apache Kafka in one operation. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.10 --- # Apache Kafka 4.10 XML and Maven Support > For the full documentation index, see: https://docs.mulesoft.com/llms.txt If you manually code a Mule app in XML either from the Anypoint Studio XML editor or from a text editor, you can get access to the Anypoint Connector for Apache Kafka (Apache Kafka Connector) in your app by adding reference statements to your XML Mule flow and in the Apache Maven `pom.xml` file. > [!NOTE] You can avoid this manual process by using Anypoint Studio. When you add the Apache Kafka > Connector in Studio, Studio automatically populates the XML code with the Apache Kafka Connector > namespace and schema location, and adds a dependency to the pom.xml file. ## Add a Namespace for the Connector Paste the following into the header of your configuration XML, inside the `` tag: ```xml http://www.mulesoft.org/schema/mule/kafka http://www.mulesoft.org/schema/mule/kafka/current/mule-kafka.xsd ``` This example shows how the namespace statements are placed in the `` XML block: ```xml ``` ## Add a POM File Dependency The Apache Maven `pom.xml` file generated by Anypoint Studio contains dependencies for a Mule app. If you code a Mule app manually, include this XML snippet in your `pom.xml` file to enable access to this connector: ```xml com.mulesoft.connectors mule-kafka-connector x.x.x mule-plugin ``` Replace `x.x.x` with the version that corresponds to the connector you are using. To obtain the most up-to-date `pom.xml` file information, access the connector in [Anypoint Exchange](https://www.mulesoft.com/exchange/) and click **Dependency Snippets**. ## Next After completing your namespace and `pom.xml` file, you can try out the Apache Kafka [Examples](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.11 --- # Apache Kafka 4.11 Examples > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following examples shows how to configure Apache Kafka Connector: - [Publish and Consume Example](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-publish-consume-example) Use the **Publish** and **Consume** to publish a message to Apache Kafka and then retrieve it. - [Integrate Confluent Schema Registry](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-configure-confluent-example) Configure Confluent Schema Registry via the **Additional Properties** field using Kafka Connector. - [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-bulk-publish-example) Use **Bulk Publish** to publish multiple messages to Apache Kafka in one operation. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.11 --- # Apache Kafka 4.11 XML and Maven Support > For the full documentation index, see: https://docs.mulesoft.com/llms.txt If you manually code a Mule app in XML either from the Anypoint Studio XML editor or from a text editor, you can get access to the Anypoint Connector for Apache Kafka (Apache Kafka Connector) in your app by adding reference statements to your XML Mule flow and in the Apache Maven `pom.xml` file. > [!NOTE] You can avoid this manual process by using Anypoint Studio. When you add the Apache Kafka > Connector in Studio, Studio automatically populates the XML code with the Apache Kafka Connector > namespace and schema location, and adds a dependency to the pom.xml file. ## Add a Namespace for the Connector Paste the following into the header of your configuration XML, inside the `` tag: ```xml http://www.mulesoft.org/schema/mule/kafka http://www.mulesoft.org/schema/mule/kafka/current/mule-kafka.xsd ``` This example shows how the namespace statements are placed in the `` XML block: ```xml ``` ## Add a POM File Dependency The Apache Maven `pom.xml` file generated by Anypoint Studio contains dependencies for a Mule app. If you code a Mule app manually, include this XML snippet in your `pom.xml` file to enable access to this connector: ```xml com.mulesoft.connectors mule-kafka-connector x.x.x mule-plugin ``` Replace `x.x.x` with the version that corresponds to the connector you are using. To obtain the most up-to-date `pom.xml` file information, access the connector in [Anypoint Exchange](https://www.mulesoft.com/exchange/) and click **Dependency Snippets**. ## Next After completing your namespace and `pom.xml` file, you can try out the Apache Kafka [Examples](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.12 --- # Apache Kafka 4.12 Examples > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following examples shows how to configure Apache Kafka Connector: - [Publish and Consume Example](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-publish-consume-example) Use the **Publish** and **Consume** to publish a message to Apache Kafka and then retrieve it. - [Integrate Confluent Schema Registry](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-configure-confluent-example) Configure Confluent Schema Registry via the **Additional Properties** field using Kafka Connector. - [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-bulk-publish-example) Use **Bulk Publish** to publish multiple messages to Apache Kafka in one operation. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.12 --- # Apache Kafka 4.12 XML and Maven Support > For the full documentation index, see: https://docs.mulesoft.com/llms.txt If you manually code a Mule app in XML either from the Anypoint Studio XML editor or from a text editor, you can get access to the Anypoint Connector for Apache Kafka (Apache Kafka Connector) in your app by adding reference statements to your XML Mule flow and in the Apache Maven `pom.xml` file. > [!NOTE] You can avoid this manual process by using Anypoint Studio. When you add the Apache Kafka > Connector in Studio, Studio automatically populates the XML code with the Apache Kafka Connector > namespace and schema location, and adds a dependency to the pom.xml file. ## Add a Namespace for the Connector Paste the following into the header of your configuration XML, inside the `` tag: ```xml http://www.mulesoft.org/schema/mule/kafka http://www.mulesoft.org/schema/mule/kafka/current/mule-kafka.xsd ``` This example shows how the namespace statements are placed in the `` XML block: ```xml ``` ## Add a POM File Dependency The Apache Maven `pom.xml` file generated by Anypoint Studio contains dependencies for a Mule app. If you code a Mule app manually, include this XML snippet in your `pom.xml` file to enable access to this connector: ```xml com.mulesoft.connectors mule-kafka-connector x.x.x mule-plugin ``` Replace `x.x.x` with the version that corresponds to the connector you are using. To obtain the most up-to-date `pom.xml` file information, access the connector in [Anypoint Exchange](https://www.mulesoft.com/exchange/) and click **Dependency Snippets**. ## Next After completing your namespace and `pom.xml` file, you can try out the Apache Kafka [Examples](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.13 --- # Apache Kafka 4.13 Examples > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following examples shows how to configure Apache Kafka Connector: - [Publish and Consume Example](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-publish-consume-example) Use the **Publish** and **Consume** to publish a message to Apache Kafka and then retrieve it. - [Integrate Confluent Schema Registry](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-configure-confluent-example) Configure Confluent Schema Registry via the **Additional Properties** field using Kafka Connector. - [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-bulk-publish-example) Use **Bulk Publish** to publish multiple messages to Apache Kafka in one operation. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.13 --- # Apache Kafka 4.13 XML and Maven Support > For the full documentation index, see: https://docs.mulesoft.com/llms.txt If you manually code a Mule app in XML either from the Anypoint Studio XML editor or from a text editor, you can get access to the Anypoint Connector for Apache Kafka (Apache Kafka Connector) in your app by adding reference statements to your XML Mule flow and in the Apache Maven `pom.xml` file. > [!NOTE] You can avoid this manual process by using Anypoint Studio. When you add the Apache Kafka > Connector in Studio, Studio automatically populates the XML code with the Apache Kafka Connector > namespace and schema location, and adds a dependency to the pom.xml file. ## Add a Namespace for the Connector Paste the following into the header of your configuration XML, inside the `` tag: ```xml http://www.mulesoft.org/schema/mule/kafka http://www.mulesoft.org/schema/mule/kafka/current/mule-kafka.xsd ``` This example shows how the namespace statements are placed in the `` XML block: ```xml ``` ## Add a POM File Dependency The Apache Maven `pom.xml` file generated by Anypoint Studio contains dependencies for a Mule app. If you code a Mule app manually, include this XML snippet in your `pom.xml` file to enable access to this connector: ```xml com.mulesoft.connectors mule-kafka-connector x.x.x mule-plugin ``` Replace `x.x.x` with the version that corresponds to the connector you are using. To obtain the most up-to-date `pom.xml` file information, access the connector in [Anypoint Exchange](https://www.mulesoft.com/exchange/) and click **Dependency Snippets**. ## Next After completing your namespace and `pom.xml` file, you can try out the Apache Kafka [Examples](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.14 --- # Apache Kafka 4.14 Examples > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following examples shows how to configure Apache Kafka Connector: - [Publish and Consume Example](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-publish-consume-example) Use the **Publish** and **Consume** to publish a message to Apache Kafka and then retrieve it. - [Integrate Confluent Schema Registry](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-configure-confluent-example) Configure Confluent Schema Registry via the **Additional Properties** field using Kafka Connector. - [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-bulk-publish-example) Use **Bulk Publish** to publish multiple messages to Apache Kafka in one operation. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.14 --- # Apache Kafka 4.14 XML and Maven Support > For the full documentation index, see: https://docs.mulesoft.com/llms.txt If you manually code a Mule app in XML either from the Anypoint Studio XML editor or from a text editor, you can get access to the Anypoint Connector for Apache Kafka (Apache Kafka Connector) in your app by adding reference statements to your XML Mule flow and in the Apache Maven `pom.xml` file. > [!NOTE] You can avoid this manual process by using Anypoint Studio. When you add the Apache Kafka > Connector in Studio, Studio automatically populates the XML code with the Apache Kafka Connector > namespace and schema location, and adds a dependency to the pom.xml file. ## Add a Namespace for the Connector Paste the following into the header of your configuration XML, inside the `` tag: ```xml http://www.mulesoft.org/schema/mule/kafka http://www.mulesoft.org/schema/mule/kafka/current/mule-kafka.xsd ``` This example shows how the namespace statements are placed in the `` XML block: ```xml ``` ## Add a POM File Dependency The Apache Maven `pom.xml` file generated by Anypoint Studio contains dependencies for a Mule app. If you code a Mule app manually, include this XML snippet in your `pom.xml` file to enable access to this connector: ```xml com.mulesoft.connectors mule-kafka-connector x.x.x mule-plugin ``` Replace `x.x.x` with the version that corresponds to the connector you are using. To obtain the most up-to-date `pom.xml` file information, access the connector in [Anypoint Exchange](https://www.mulesoft.com/exchange/) and click **Dependency Snippets**. ## Next After completing your namespace and `pom.xml` file, you can try out the Apache Kafka [Examples](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Apache Kafka 4.15 Examples > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following examples shows how to configure Apache Kafka Connector: - [Publish and Consume Example](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-publish-consume-example) Use the **Publish** and **Consume** to publish a message to Apache Kafka and then retrieve it. - [Integrate Confluent Schema Registry](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-configure-confluent-example) Configure Confluent Schema Registry via the **Additional Properties** field using Kafka Connector. - [Configure AWS IAM Authentication](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-configure-aws-iam-example) Configure SASL/AWS IAM authentication to connect to an Amazon MSK cluster. - [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-bulk-publish-example) Use **Bulk Publish** to publish multiple messages to Apache Kafka in one operation. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Apache Kafka 4.15 XML and Maven Support > For the full documentation index, see: https://docs.mulesoft.com/llms.txt If you manually code a Mule app in XML either from the Anypoint Studio XML editor or from a text editor, you can get access to the Anypoint Connector for Apache Kafka (Apache Kafka Connector) in your app by adding reference statements to your XML Mule flow and in the Apache Maven `pom.xml` file. > [!NOTE] You can avoid this manual process by using Anypoint Studio. When you add the Apache Kafka > Connector in Studio, Studio automatically populates the XML code with the Apache Kafka Connector > namespace and schema location, and adds a dependency to the pom.xml file. ## Add a Namespace for the Connector Paste the following into the header of your configuration XML, inside the `` tag: ```xml http://www.mulesoft.org/schema/mule/kafka http://www.mulesoft.org/schema/mule/kafka/current/mule-kafka.xsd ``` This example shows how the namespace statements are placed in the `` XML block: ```xml ``` ## Add a POM File Dependency The Apache Maven `pom.xml` file generated by Anypoint Studio contains dependencies for a Mule app. If you code a Mule app manually, include this XML snippet in your `pom.xml` file to enable access to this connector: ```xml com.mulesoft.connectors mule-kafka-connector x.x.x mule-plugin ``` Replace `x.x.x` with the version that corresponds to the connector you are using. To obtain the most up-to-date `pom.xml` file information, access the connector in [Anypoint Exchange](https://www.mulesoft.com/exchange/) and click **Dependency Snippets**. ## Next After completing your namespace and `pom.xml` file, you can try out the Apache Kafka [Examples](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.10 --- # Apache Kafka Connector 4.10 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system and achieve seamless integration between your Mule app and a Kafka cluster, using Mule runtime engine (Mule). For compatibility information, see the Apache Kafka Connector Release Notes. ## Before You Begin Before creating an app, you must: - Have access to the Apache Kafka target resource and Anypoint Platform - Understand how to create a Mule app using Anypoint Studio - Have access to Apache Kafka to get values for the fields that appear in Studio ## Common Use Cases for the Connector Common use cases for Apache Kafka Connector include: - **Log Aggregation** Leverage the low-latency processing from Apache Kafka to collect logs from multiple services and make them available in a standard format to multiple consumers. For example, you can publish a message to Apache Kafka and then retrieve it. - **Analysis and Metrics** Optimize your advertising budget by integrating Apache Kafka and your big data analytics solution to analyze end user activity, such as page views, clicks, shares, and so on, to serve relevant ads. For example, a department store can use a website activity tracker to improve the online shopping experience. The gathered data is sent to multiple departments for various computations. Each department reviews the received information to stay informed about what the customer is looking for and then provides recommendations accordingly. - **Notifications and Alerts** Notify customers about various financial events, such as the recent amount of a transaction or more complex events like future investment suggestions based on integrations with credit agencies, location, and so on. - **Time-Sensitive Applications** Build integration applications that are time-sensitive. For example, you can build an application that ensures that patients in a hospital receive the care they need in a timely manner with a hospital server that processes emergency requests for patient admissions and orders the requests based on priority determined by specified criteria. In this scenario, an application processes Apache Kafka messages in the order that they are received, relying on the order and idempotency of the messages sent through the queue. For example, a newsroom can use the Apache Kafka system to deliver the latest news. Retrieve the latest news by building an integration application by reading from the end of the Apache Kafka queue first. For examples of these use cases, refer to [Apache Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-examples). ### Publish-to Use Cases With Kafka Connector, applications can publish binary messages to the Kafka server. Some examples of use cases include: - Publishing binary data to the Kafka server and the underlying topic. - Setting the batch size of the data to publish to Kafka when working with large volumes of data. - To optimize for performance, companies can batch messages to send. Kafka stores the message until the batch size is reached and completes the send at that time. - Configuring the message size to send to focus on optimal performance. - Kafka is not intended to process and handle large messages. Messages are typically in the 1 MB range and users set the size to avoid performance degradation. In the case of an error, the message is rejected. - Blocking further commits on a configurable level of acknowledgment for the messages published to the Kafka server to ensure successful data transmission. - Acknowledgments enable users to ensure that the message is sent prior to publishing more messages. This is specifically useful in time-ordered or event-ordered messages, such as logs or patient drug administration orders. Ensuring that messages are sent in order helps the downstream application process the messages in the right order. - Configuring how data is serialized during a publish. - Securing messages sent during Mutual TLS. The following lists the acceptance criteria for publishing: - Publish a message to a Kafka topic. - Configure the batch size and size of the message. - Obtain an acknowledgment once publishing is complete at the configured level. - Send the message securely via Mutual TLS. - Configure the serialization type. ### Subscribe-from Use Cases With Kafka Connector, applications can obtain messages from Kafka for further consumption downstream. Some examples of use cases include: Reading from Kafka: - Subscribing to the Kafka server and the underlying group to read and consume messages for downstream processing. - Subscribing to a specific Kafka partition to process messages. - Users with a high availability microservice don’t want to rely on Kafka’s rebalancing in the event of a failure. - If users have a local state associated with their partition, they want to process only that partition. - Configuring the offset value to a set value or to the beginning or end of the topic. - Users have a local store and want to use that information to start processing their records. - Users have time-sensitive records and want to skip to the end if the application must be up-to-date or skip to the beginning if it is a new application processing the records. - Reading the offset value from an alternate storage system on restart. - Users have a local store with the offset information and want to read the value from their local storage. - Setting the interval for which to block the receipt of messages for varying downstream processing intervals. Processing the messages: - Ensuring there is no data loss when the downstream application crashes by manually configuring commits to have more control. - Storing a commit value in a local store or in the Kafka storage and blocking until the store is successful. - Storing an offset asynchronously in a local store or in the Kafka storage. - Using Kafka’s in-built capability of load-balancing with multiple applications reading from the same group ID. - Configuring the number of records to process in a single call. The following lists the acceptance criteria for subscribing: - Read a message from Kafka from a specific partition or topic. - Read a message from Kafka after obtaining the offset from a local store. - Read a message from the beginning of the queue, the end of the queue, and a pre-specified offset. - Ensure a high level of performance when there is one consumer per thread. Multiple consumers exist in a single group. ## Audience - Starting user To create a Mule app, see [Apache Kafka Studio Configuration](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-studio). - Power user Read [XML and Maven Support](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-xml-maven) and [Examples](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-examples). ## Next After you complete the prerequisites, you are ready to create an app with [Anypoint Studio](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-studio). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Use Exchange to Discover Connectors, Templates, and Examples](https://docs.mulesoft.com/connectors/introduction/intro-use-exchange) - [Apache Kafka Connector](https://www.mulesoft.com/exchange/com.mulesoft.connectors/mule-kafka-connector/) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.10 --- # Apache Kafka Connector 4.10 - Additional Configuration Information > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Set additional configuration for Anypoint Connector for Apache Kafka (Apache Kafka Connector). ## Configure a Custom Partitioner You can configure a custom partitioner for Apache Kafka Connector in a Mule app: 1. Use Maven to create a Java project and add the custom partitioner to it: ```java public class CustomPartitioner implements Partitioner { // Implementation } ``` 2. Add this dependency in the Mule app: ```java org.example custom-partitioner 1.0-SNAPSHOT ``` 3. Add this dependency to `additionalPluginDependencies`: ```java org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 org.example custom-partitioner 1.0-SNAPSHOT mule-application ``` 4. Add the custom partitioner to the producer configuration: ```java ``` ## Next Step After you complete configuring the connector, you can try the [examples](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.10 --- # Apache Kafka Connector 4.10 Reference > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system. It provides seamless integration between your Mule app and an Apache Kafka cluster, using Mule runtime engine (Mule). > [!NOTE] When a **Message Listener** or a **Batch Message Listener** uses the `MANUAL` > acknowledgement mode, you must use a **Commit** operation at the end of the flow when the flow > finishes successfully (either because there were no errors or because there is an **On Error > Continue** component in the flow). > [!NOTE] Mule server notifications are not supported because the Kafka library used in the > connector manages network disruptions internally. ## Consumer Configuration Configuration for consumers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default acknowledgement mode

Enumeration, one of

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

AUTO

Default listener poll timeout

Number

Time to wait to perform a poll if data is not available in the buffer (fetched). If no value is set, the poll is returned immediately with any records that are currently available in the buffer or else returns empty if there is no data. Must not be negative.

100

Default listener poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default listener poll timeout field. This combines with Poll Timeout to define the total timeout for the polling.

MILLISECONDS

Default operation poll timeout

Number

Time to wait for an operation to finish. If no value is set or a negative value is set, the operation waits forever.

-1

Default operation poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default operation timeout field. This combines with Operation Timeout to define the total default timeout for the operations that use this configuration.

SECONDS

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Consumer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. Additionally, the connector supports only [RFC6749 standard](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4.3). For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Group ID

String

Default Group ID for all the Kafka Consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Whether internal topics matching a subscribed pattern is excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the reconnect backoff timeout scalar.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

true

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this field at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. You can override this field at the source level.

KB

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this field at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Request Timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This parameter can be overridden at source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Default record limit

Number

The maximum number of records returned on a poll call to the Kafka cluster. This parameter can be overridden at source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

The expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch heartbeat interval time scalar.

SECONDS

Session timeout

Number

The timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for session timeout scalar.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this config.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for connections maximum idle time scalar.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Topic Subscription Patterns

Array of String

The list of subscription regular expressions to subscribe to. This topics will be automatically rebalanced between the amount of consumers of the topic.

Assignments

The list of topic-partition pairs to assign. Note that there will be no automatic rebalance of the consumers

Default fetch minimum size

Number

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request waits for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency. This parameter can be overridden at source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

BYTE

Default fetch maximum size

Number

The maximum amount of data the server should return for a fetch request. Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). Note that the consumer performs multiple fetches in parallel. This parameter can be overridden at source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Default maximum partition fetch size

Number

The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size.This parameter can be overridden at source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Fetch Maximum Wait Timeout

Number

The maximum amount of time the server will block before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch maximum wait timeout scalar.

MILLISECONDS

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Consumer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Encryption type

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

User used by the client to connect to the Kafka broker.

x

Password

String

Password used by the client to connect to the Kafka broker.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

--- ## Producer Configuration Configuration for producers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default topic

String

Default topic name to use by the producer operations, overridable at the operation’s configuration level.

defaultTopicName

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Producer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names..

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Batch size

Number

The producer attempts to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size. Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent. A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the batch size scalar.

KB

Buffer size

Number

The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block for max.block.ms after which it will throw an exception. This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests. The default value in the Kafka docs is of 33554432 (32MB), but this should be capped to align with expected values for mule instances in cloudhub (v0.1 core)

1000

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

KB

DNS lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).

NONE

Connections maximum idle time

Number

Close idle connections after the value specified by this config.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the connections maximum idle scalar.

SECONDS

Delivery timeout

Number

An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retriable send failures. The producer may report failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch which reached an earlier delivery expiration deadline. The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the delivery timeout scalar.

SECONDS

Enable idempotence

Boolean

When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc, may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0 and acks must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConnectionException will be thrown

false

Linger time

Number

The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay?that is, rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it is sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the linger time scalar.

SECONDS

Maximum block time

Number

The configuration controls how long KafkaProducer.send() and KafkaProducer.partitionsFor() will block.These methods can be blocked either because the buffer is full or metadata unavailable.Blocking in the user-supplied serializers or partitioner will not be counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the maximum block time scalar.

SECONDS

Maximum in flight requests

Number

The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).

5

Maximum request size

Number

The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size which may be different from this.

1

Maximum request size unit.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

MB

Producer acknowledge mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent

NONE

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. This parameter can be overridden at source level.

KB

Retries amount

Number

Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first. Note additionally that produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires first before successful acknowledgment. Users should generally prefer to leave this config unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the retry backoff timeout time scalar.

MILLISECONDS

Retry backoff timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the request timeout time scalar.

SECONDS

Default request timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This should be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

The configuration controls which partitioning strategy is used when sending messages without providing a key or a partition.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Producer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

EncryptionType

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

## Consumer Sources - [Batch Message Listener](#batch-message-listener) - [Message Listener](#message-listener) ### Batch Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a list of messages to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Record

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) > [!NOTE] The **Batch Message Listener** source does not support configurable streaming strategies > because the source takes a batch of records as an input stream. The streaming strategy > configuration is non-repeatable-stream by default for the source. ### Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a single message to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) ## Consumer Operations - [Commit](#commit) - [Consume](#consume) - [Seek](#seek) ### Commit `` Commits the offsets associated to a message or batch of messages consumed in a **Message Listener**. This is a list or a single message consumed in the **Batch Message Listener** source.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumer commit key

String

Commit key of the last poll. This operation is valid only when used inside a flow that uses one of the sources (Batch Message Listener or Message Listener) which inserts this value as an attribute in the Mule event.

x

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:INVALID\_ACK\_MODE - KAFKA:RETRY\_EXHAUSTED - KAFKA:ALREADY\_COMMITTED - KAFKA:TIMEOUT - KAFKA:SESSION\_NOT\_FOUND - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Consume `` Receives messages from one or more Kafka topics. This operation works similarly to the **Message Listener** source, so all the operations that apply to that, apply to this operation as well. > [!NOTE] The **Consume** operation does not return the `consumerCommitKey`.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumption timeout

Number

Number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Consumption timeout field.

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

IMMEDIATE

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:RETRY\_EXHAUSTED - KAFKA:ILLEGAL\_STATE - KAFKA:TIMEOUT - KAFKA:SERIALIZATION\_ERROR - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Seek `` Sets the current offset of the consumer for the given topic and partition to the provided offset value.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which the Seek operation is performed.

x

Partition

Number

Partition number that has its offset modified.

x

Offset

Number

Offset value to commit for the configured partition.

x

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:INVALID\_TOPIC - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ## Producer Operation - [Bulk Publish](#bulkPublish) - [Publish](#publish) ### Bulk Publish `` Publishes a bulk of messages to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Messages

Array of Kafka Message

List of messages to publish.

#[payload]

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:AUTHENTICATION\_ERROR - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ### Publish `` Publishes a message to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Topic to publish to.

Partition

Number

(Optional) Topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ## Object Types - [Consumer Context](#ConsumerContext) - [CRL File](#CrlFile) - [Custom OCSP Responder](#CustomOcspResponder) - [Expiration Policy](#ExpirationPolicy) - [Kafka Message](#KafkaMessage) - [Kafka Message Attributes](#KafkaMessageAttributes) - [Kafka Message Metadata](#KafkaMessageMetadata) - [Kafka Record Attributes](#KafkaRecordAttributes) - [Keystore](#KeyStore) - [Proxy Config Params](#ProxyConfigParams) - [Reconnect](#Reconnect) - [Reconnect Forever](#Reconnect-Forever) - [Reconnection](#Reconnection) - [Record](#Record) - [Redelivery Policy](#RedeliveryPolicy) - [Repeatable File Store Stream](#RepeatableFileStoreStream) - [Repeatable In Memory Stream](#RepeatableInMemoryStream) - [Standard Revocation Check](#StandardRevocationCheck) - [Tcp Client Socket Params](#TcpClientSocketParams) - [TLS](#Tls) - [Topic Partition](#TopicPartition) - [Truststore](#TrustStore) ### Consumer Context Type for the consumer context.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

### CRL File Specifies the location of the certification revocation list (CRL) file.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the CRL file.

### Custom OCSP Responder Configures a custom OCSP responder for certification revocation checks.
FieldTypeDescriptionDefault ValueRequired

Url

String

URL of the OCSP responder.

Cert Alias

String

Alias of the signing certificate for the OCSP response. If specified, the alias must be in the truststore.

### Expiration Policy Configures an expiration policy strategy.
FieldTypeDescriptionDefault ValueRequired

Max Idle Time

Number

Configures the maximum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Max Idle Time field.

### Kafka Message Represents a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Attributes

Attributes associated with the Kafka message.

Payload

Binary

Payload of the Kafka message, represented as an InputStream.

### Kafka Message Attributes Represents the attributes of a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Headers

Object

Headers of the Kafka message, represented as a map in which the key is a string and the value is an InputStream.

Key

Binary

Key of the Kafka message, represented as an InputStream.

Partition

Number

Partition of the topic where the Kafka message is sent.

Topic

String

Topic to which the Kafka message belongs to.

### Kafka Message Metadata Metadata of the Kafka message.
FieldTypeDescriptionDefault ValueRequired

Offset

Number

Offset of the message.

Partition

Number

Partition of the message.

Serialized Key Size

Number

Serialized key size of the message.

Serialized Value Size

Number

Serialized value size of the message.

Timestamp

DateTime

Timestamp of the message.

Topic

String

Topic of the message.

### Kafka Record Attributes Attributes of the Kafka record.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

Creation Timestamp

DateTime

Timestamp of record creation.

Headers

Object

Map of HTTP headers in the message.

Key

Binary

Keys of the record.

Leader Epoch

Number

Leader epoch of the record.

Log Append Timestamp

DateTime

Log append timestamp of the record.

Offset

Number

Offset of the record.

Partition

Number

Partition of the record.

Serialized Key Size

Number

Serialized key size of the record.

Serialized Value Size

Number

Serialized value size of the record.

Topic

String

Topic of the record.

### Keystore Configures the keystore for the TLS protocol. The keystore you generate contains a private key and a public certificate.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the keystore. Mule resolves the path relative to the current classpath and file system.

Type

String

Type of keystore.

Alias

String

Alias of the key to use when the keystore contains multiple private keys. By default, Mule uses the first key in the file.

Key Password

String

Password used to protect the private key.

Password

String

Password used to protect the keystore.

Algorithm

String

Encryption algorithm that the keystore uses.

### Proxy Config Params Configures the proxy configuration parameters.
FieldTypeDescriptionDefault ValueRequired

Host

String

Hostname or IP address of the proxy server.

Port

Number

Port of the proxy server.

Username

String

Username to authenticate against the proxy server.

Password

String

Password to authenticate against the proxy server.

### Reconnect Configures a standard reconnection strategy, which specifies how often to reconnect and how many reconnection attempts the connector source or operation can make.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

Count

Number

How many reconnection attempts the Mule app can make.

### Reconnect Forever Configures a forever reconnection strategy by which the connector source or operation attempts to reconnect at a specified frequency for as long as the Mule app runs.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

### Reconnection Configures a reconnection strategy for an operation.
FieldTypeDescriptionDefault ValueRequired

Fails Deployment

Boolean

What to do if, when an app is deployed, a connectivity test does not pass after exhausting the associated reconnection strategy:

  • true

    Allow the deployment to fail.

  • false

    Ignore the results of the connectivity test.

Reconnection Strategy

Reconnection strategy to use.

### Record Type for records.
FieldTypeDescriptionDefault ValueRequired

Attributes

Kafka record attributes.

Payload

Binary

Payload for the record.

### Redelivery Policy Configures the redelivery policy for executing requests that generate errors. You can add a redelivery policy to any source in a flow.
FieldTypeDescriptionDefault ValueRequired

Max Redelivery Count

Number

Maximum number of times that a redelivered request can be processed unsuccessfully before returning a REDELIVERY_EXHAUSTED error.

Use Secure Hash

Boolean

If true, Mule uses a secure hash algorithm to identify a redelivered message.

Message Digest Algorithm

String

Secure hashing algorithm to use if the Use Secure Hash field is true. If the payload of the message is a Java object, Mule ignores this value and returns the value that the payload’s hashCode() returned.

Id Expression

String

Defines one or more expressions to use to determine when a message has been redelivered. This property may only be set if Use Secure Hash is false.

Object Store

ObjectStore

Configures the object store that stores the redelivery counter for each message.

### Repeatable File Store Stream Configures the repeatable file-store streaming strategy by which Mule keeps a portion of the stream content in memory. If the stream content is larger than the configured buffer size, Mule backs up the buffer’s content to disk and then clears the memory.
FieldTypeDescriptionDefault ValueRequired

In Memory Size

Number

Maximum amount of memory that the stream can use for data. If the amount of memory exceeds this value, Mule buffers the content to disk. To optimize performance:

  • Configure a larger buffer size to avoid the number of times Mule needs to write the buffer on disk. This increases performance, but it also limits the number of concurrent requests your application can process, because it requires additional memory.

  • Configure a smaller buffer size to decrease memory load at the expense of response time.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the In Memory Size field.

### Repeatable In Memory Stream Configures the in-memory streaming strategy by which the request fails if the data exceeds the MAX buffer size. Always run performance tests to find the optimal buffer size for your specific use case.
FieldTypeDescriptionDefault ValueRequired

Initial Buffer Size

Number

Initial amount of memory to allocate to the data stream. If the streamed data exceeds this value, the buffer expands by Buffer Size Increment, with an upper limit of Max In Memory Size value.

Buffer Size Increment

Number

Amount by which the buffer size expands if it exceeds its initial size. Setting a value of 0 or lower specifies that the buffer can’t expand.

Max Buffer Size

Number

Maximum amount of memory to use. If more than that is used then a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value lower than or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the Initial Buffer Size, Buffer Size Increment, and Buffer Unit fields.

### Standard Revocation Check Configures standard revocation checks for TLS certificates.
FieldTypeDescriptionDefault ValueRequired

Only End Entities

Boolean

Which elements to verify in the certificate chain:

  • true

    Verify only the last element in the certificate chain.

  • false

    Verify all elements in the certificate chain.

Prefer Crls

Boolean

How to check certificate validity:

  • true

    Check the Certification Revocation List (CRL) for certificate validity.

  • false

    Use the Online Certificate Status Protocol (OCSP) to check certificate validity.

No Fallback

Boolean

Whether to use the secondary method to check certificate validity:

  • true

    Use the method that wasn’t specified in the Prefer Crls field (the secondary method) to check certificate validity.

  • false

    Do not use the secondary method to check certificate validity.

Soft Fail

Boolean

What to do if the revocation server can’t be reached or is busy:

  • true

    Avoid verification failure.

  • false

    Allow the verification to fail.

### Tcp Client Socket Params Configures the TCP client socket parameters.
FieldTypeDescriptionDefault ValueRequired

Send Buffer Size

Number

Size of the buffer (in bytes) used when sending data, set on the socket itself.

Receive Buffer Size

Number

Buffer size.

Client Timeout

Number

Sets the SO_TIMEOUT value on sockets. Indicates the amount of time (in milliseconds) that the socket waits in a blocking operation before failing. A value of 0 (the default) means waiting indefinitely.

0

Send Tcp No Delay

Boolean

If set, data is sent immediately rather than being collected for efficiency, prioritizing latency over network traffic reduction. Despite the socket default being false, this option defaults to true because reducing network traffic is generally not a major concern.

true

Linger

Number

Sets the SO_LINGER value in milliseconds. This determines how long the socket remains open to ensure all remaining data is transmitted correctly before closing.

Keep Alive

Boolean

Enables the SO_KEEPALIVE behavior on open sockets. This option periodically checks idle socket connections and closes them if they become unavailable. It is used by server sockets to control whether connections are kept alive before being recycled.

false

Connection Timeout

Number

How long the connector waits before timing out when establishing a connection to the remote service.

30000

### TLS Configures TLS to provide secure communications for the Mule app.
FieldTypeDescriptionDefault ValueRequired

Enabled Protocols

String

Comma-separated list of protocols enabled for this context.

Enabled Cipher Suites

String

Comma-separated list of cipher suites enabled for this context.

Trust Store

Configures the TLS truststore.

Key Store

Configures the TLS keystore.

Revocation Check

Configures a revocation checking mechanism.

### Topic Partition Type for topic partition.
FieldTypeDescriptionDefault ValueRequired

Topic

String

Topic name.

x

Partition

Number

Number of partitions.

x

### Truststore Configures the truststore for TLS.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the truststore. Mule resolves the path relative to the current classpath and file system.

Password

String

Password used to protect the truststore.

Type

String

Type of truststore.

Algorithm

String

Encryption algorithm that the truststore uses.

Insecure

Boolean

If true, Mule stops performing certificate validations. Setting this to true can make connections vulnerable to attacks.

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.11 --- # Apache Kafka Connector 4.11 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system and achieve seamless integration between your Mule app and a Kafka cluster, using Mule runtime engine (Mule). For compatibility information, see the Apache Kafka Connector Release Notes. ## Before You Begin Before creating an app, you must: - Have access to the Apache Kafka target resource and Anypoint Platform - Understand how to create a Mule app using Anypoint Studio - Have access to Apache Kafka to get values for the fields that appear in Studio ## Common Use Cases for the Connector Common use cases for Apache Kafka Connector include: - **Log Aggregation** Leverage the low-latency processing from Apache Kafka to collect logs from multiple services and make them available in a standard format to multiple consumers. For example, you can publish a message to Apache Kafka and then retrieve it. - **Analysis and Metrics** Optimize your advertising budget by integrating Apache Kafka and your big data analytics solution to analyze end user activity, such as page views, clicks, shares, and so on, to serve relevant ads. For example, a department store can use a website activity tracker to improve the online shopping experience. The gathered data is sent to multiple departments for various computations. Each department reviews the received information to stay informed about what the customer is looking for and then provides recommendations accordingly. - **Notifications and Alerts** Notify customers about various financial events, such as the recent amount of a transaction or more complex events like future investment suggestions based on integrations with credit agencies, location, and so on. - **Time-Sensitive Applications** Build integration applications that are time-sensitive. For example, you can build an application that ensures that patients in a hospital receive the care they need in a timely manner with a hospital server that processes emergency requests for patient admissions and orders the requests based on priority determined by specified criteria. In this scenario, an application processes Apache Kafka messages in the order that they are received, relying on the order and idempotency of the messages sent through the queue. For example, a newsroom can use the Apache Kafka system to deliver the latest news. Retrieve the latest news by building an integration application by reading from the end of the Apache Kafka queue first. For examples of these use cases, refer to [Apache Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-examples). ### Publish-to Use Cases With Kafka Connector, applications can publish binary messages to the Kafka server. Some examples of use cases include: - Publishing binary data to the Kafka server and the underlying topic. - Setting the batch size of the data to publish to Kafka when working with large volumes of data. - To optimize for performance, companies can batch messages to send. Kafka stores the message until the batch size is reached and completes the send at that time. - Configuring the message size to send to focus on optimal performance. - Kafka is not intended to process and handle large messages. Messages are typically in the 1 MB range and users set the size to avoid performance degradation. In the case of an error, the message is rejected. - Blocking further commits on a configurable level of acknowledgment for the messages published to the Kafka server to ensure successful data transmission. - Acknowledgments enable users to ensure that the message is sent prior to publishing more messages. This is specifically useful in time-ordered or event-ordered messages, such as logs or patient drug administration orders. Ensuring that messages are sent in order helps the downstream application process the messages in the right order. - Configuring how data is serialized during a publish. - Securing messages sent during Mutual TLS. The following lists the acceptance criteria for publishing: - Publish a message to a Kafka topic. - Configure the batch size and size of the message. - Obtain an acknowledgment once publishing is complete at the configured level. - Send the message securely via Mutual TLS. - Configure the serialization type. ### Subscribe-from Use Cases With Kafka Connector, applications can obtain messages from Kafka for further consumption downstream. Some examples of use cases include: Reading from Kafka: - Subscribing to the Kafka server and the underlying group to read and consume messages for downstream processing. - Subscribing to a specific Kafka partition to process messages. - Users with a high availability microservice don’t want to rely on Kafka’s rebalancing in the event of a failure. - If users have a local state associated with their partition, they want to process only that partition. - Configuring the offset value to a set value or to the beginning or end of the topic. - Users have a local store and want to use that information to start processing their records. - Users have time-sensitive records and want to skip to the end if the application must be up-to-date or skip to the beginning if it is a new application processing the records. - Reading the offset value from an alternate storage system on restart. - Users have a local store with the offset information and want to read the value from their local storage. - Setting the interval for which to block the receipt of messages for varying downstream processing intervals. Processing the messages: - Ensuring there is no data loss when the downstream application crashes by manually configuring commits to have more control. - Storing a commit value in a local store or in the Kafka storage and blocking until the store is successful. - Storing an offset asynchronously in a local store or in the Kafka storage. - Using Kafka’s in-built capability of load-balancing with multiple applications reading from the same group ID. - Configuring the number of records to process in a single call. The following lists the acceptance criteria for subscribing: - Read a message from Kafka from a specific partition or topic. - Read a message from Kafka after obtaining the offset from a local store. - Read a message from the beginning of the queue, the end of the queue, and a pre-specified offset. - Ensure a high level of performance when there is one consumer per thread. Multiple consumers exist in a single group. ## Audience - Starting user To create a Mule app, see [Apache Kafka Studio Configuration](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-studio). - Power user Read [XML and Maven Support](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-xml-maven) and [Examples](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-examples). ## Next After you complete the prerequisites, you are ready to create an app with [Anypoint Studio](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-studio). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Use Exchange to Discover Connectors, Templates, and Examples](https://docs.mulesoft.com/connectors/introduction/intro-use-exchange) - [Apache Kafka Connector](https://www.mulesoft.com/exchange/com.mulesoft.connectors/mule-kafka-connector/) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.11 --- # Apache Kafka Connector 4.11 - Additional Configuration Information > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Set additional configuration for Anypoint Connector for Apache Kafka (Apache Kafka Connector). ## Configure a Custom Partitioner You can configure a custom partitioner for Apache Kafka Connector in a Mule app: 1. Use Maven to create a Java project and add the custom partitioner to it: ```java public class CustomPartitioner implements Partitioner { // Implementation } ``` 2. Add this dependency in the Mule app: ```java org.example custom-partitioner 1.0-SNAPSHOT ``` 3. Add this dependency to `additionalPluginDependencies`: ```java org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 org.example custom-partitioner 1.0-SNAPSHOT mule-application ``` 4. Add the custom partitioner to the producer configuration: ```java ``` ## Next Step After you complete configuring the connector, you can try the [examples](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.11 --- # Apache Kafka Connector 4.11 Reference > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system. It provides seamless integration between your Mule app and an Apache Kafka cluster, using Mule runtime engine (Mule). > [!NOTE] When a **Message Listener** or a **Batch Message Listener** uses the `MANUAL` > acknowledgement mode, you must use a **Commit** operation at the end of the flow when the flow > finishes successfully (either because there were no errors or because there is an **On Error > Continue** component in the flow). > [!NOTE] Mule server notifications are not supported because the Kafka library used in the > connector manages network disruptions internally. ## Consumer Configuration Configuration for consumers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default acknowledgement mode

Enumeration, one of

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

AUTO

Default listener poll timeout

Number

Time to wait to perform a poll if data is not available in the buffer (fetched). If no value is set, the poll is returned immediately with any records that are currently available in the buffer or else returns empty if there is no data. Must not be negative.

100

Default listener poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default listener poll timeout field. This combines with Poll Timeout to define the total timeout for the polling.

MILLISECONDS

Default operation poll timeout

Number

Time to wait for an operation to finish. If no value is set or a negative value is set, the operation waits forever.

-1

Default operation poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default operation timeout field. This combines with Operation Timeout to define the total default timeout for the operations that use this configuration.

SECONDS

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Consumer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. Additionally, the connector supports only [RFC6749 standard](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4.3). For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Group ID

String

Default Group ID for all the Kafka Consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Whether internal topics matching a subscribed pattern is excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the reconnect backoff timeout scalar.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

true

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this field at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. You can override this field at the source level.

KB

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this field at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Request Timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This parameter can be overridden at source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Default record limit

Number

The maximum number of records returned on a poll call to the Kafka cluster. This parameter can be overridden at source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

The expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch heartbeat interval time scalar.

SECONDS

Session timeout

Number

The timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for session timeout scalar.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this config.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for connections maximum idle time scalar.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Topic Subscription Patterns

Array of String

The list of subscription regular expressions to subscribe to. This topics will be automatically rebalanced between the amount of consumers of the topic.

Assignments

The list of topic-partition pairs to assign. Note that there will be no automatic rebalance of the consumers

Default fetch minimum size

Number

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request waits for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency. This parameter can be overridden at source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

BYTE

Default fetch maximum size

Number

The maximum amount of data the server should return for a fetch request. Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). Note that the consumer performs multiple fetches in parallel. This parameter can be overridden at source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Default maximum partition fetch size

Number

The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size.This parameter can be overridden at source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Fetch Maximum Wait Timeout

Number

The maximum amount of time the server will block before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch maximum wait timeout scalar.

MILLISECONDS

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Consumer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Encryption type

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation. Enable FIPS mode by setting the `com.mulesoft.connectors.allow-registration-of-sasl-security-provider-in-fips=true` flag to true. This adds the `com.sun.security.sasl.Provider` security provider in the FIPS environment.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

User used by the client to connect to the Kafka broker.

x

Password

String

Password used by the client to connect to the Kafka broker.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

--- ## Producer Configuration Configuration for producers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default topic

String

Default topic name to use by the producer operations, overridable at the operation’s configuration level.

defaultTopicName

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Producer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names..

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Batch size

Number

The producer attempts to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size. Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent. A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the batch size scalar.

KB

Buffer size

Number

The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block for max.block.ms after which it will throw an exception. This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests. The default value in the Kafka docs is of 33554432 (32MB), but this should be capped to align with expected values for mule instances in cloudhub (v0.1 core)

1000

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

KB

DNS lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).

NONE

Connections maximum idle time

Number

Close idle connections after the value specified by this config.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the connections maximum idle scalar.

SECONDS

Delivery timeout

Number

An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retriable send failures. The producer may report failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch which reached an earlier delivery expiration deadline. The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the delivery timeout scalar.

SECONDS

Enable idempotence

Boolean

When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc, may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0 and acks must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConnectionException will be thrown

false

Linger time

Number

The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay?that is, rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it is sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the linger time scalar.

SECONDS

Maximum block time

Number

The configuration controls how long KafkaProducer.send() and KafkaProducer.partitionsFor() will block.These methods can be blocked either because the buffer is full or metadata unavailable.Blocking in the user-supplied serializers or partitioner will not be counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the maximum block time scalar.

SECONDS

Maximum in flight requests

Number

The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).

5

Maximum request size

Number

The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size which may be different from this.

1

Maximum request size unit.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

MB

Producer acknowledge mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent

NONE

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. This parameter can be overridden at source level.

KB

Retries amount

Number

Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first. Note additionally that produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires first before successful acknowledgment. Users should generally prefer to leave this config unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the retry backoff timeout time scalar.

MILLISECONDS

Retry backoff timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the request timeout time scalar.

SECONDS

Default request timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This should be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

The configuration controls which partitioning strategy is used when sending messages without providing a key or a partition.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Producer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

EncryptionType

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

## Consumer Sources - [Batch Message Listener](#batch-message-listener) - [Message Listener](#message-listener) ### Batch Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a list of messages to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Record

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) > [!NOTE] The **Batch Message Listener** source does not support configurable streaming strategies > because the source takes a batch of records as an input stream. The streaming strategy > configuration is non-repeatable-stream by default for the source. ### Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a single message to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) ## Consumer Operations - [Commit](#commit) - [Consume](#consume) - [Seek](#seek) ### Commit `` Commits the offsets associated to a message or batch of messages consumed in a **Message Listener**. This is a list or a single message consumed in the **Batch Message Listener** source.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumer commit key

String

Commit key of the last poll. This operation is valid only when used inside a flow that uses one of the sources (Batch Message Listener or Message Listener) which inserts this value as an attribute in the Mule event.

x

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_ACK\_MODE - KAFKA:RETRY\_EXHAUSTED - KAFKA:ALREADY\_COMMITTED - KAFKA:TIMEOUT - KAFKA:SESSION\_NOT\_FOUND - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Consume `` Receives messages from one or more Kafka topics. This operation works similarly to the **Message Listener** source, so all the operations that apply to that, apply to this operation as well. > [!NOTE] The **Consume** operation does not return the `consumerCommitKey`.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumption timeout

Number

Number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Consumption timeout field.

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

IMMEDIATE

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:SECURITY - KAFKA:RETRY\_EXHAUSTED - KAFKA:ILLEGAL\_STATE - KAFKA:TIMEOUT - KAFKA:SERIALIZATION\_ERROR - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Seek `` Sets the current offset of the consumer for the given topic and partition to the provided offset value.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which the Seek operation is performed.

x

Partition

Number

Partition number that has its offset modified.

x

Offset

Number

Offset value to commit for the configured partition.

x

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:INVALID\_TOPIC - KAFKA:SECURITY - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ## Producer Operations - [Bulk Publish](#bulkPublish) - [Publish](#publish) ### Bulk Publish `` Publishes a bulk of messages to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Messages

Array of Kafka Message

List of messages to publish.

#[payload]

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:AUTHENTICATION\_ERROR - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ### Publish `` Publishes a message to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Topic to publish to.

Partition

Number

(Optional) Topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ## Object Types - [Consumer Context](#ConsumerContext) - [CRL File](#CrlFile) - [Custom OCSP Responder](#CustomOcspResponder) - [Expiration Policy](#ExpirationPolicy) - [Kafka Message](#KafkaMessage) - [Kafka Message Attributes](#KafkaMessageAttributes) - [Kafka Message Metadata](#KafkaMessageMetadata) - [Kafka Record Attributes](#KafkaRecordAttributes) - [Keystore](#KeyStore) - [Proxy Config Params](#ProxyConfigParams) - [Reconnect](#Reconnect) - [Reconnect Forever](#Reconnect-Forever) - [Reconnection](#Reconnection) - [Record](#Record) - [Redelivery Policy](#RedeliveryPolicy) - [Repeatable File Store Stream](#RepeatableFileStoreStream) - [Repeatable In Memory Stream](#RepeatableInMemoryStream) - [Standard Revocation Check](#StandardRevocationCheck) - [Tcp Client Socket Params](#TcpClientSocketParams) - [TLS](#Tls) - [Topic Partition](#TopicPartition) - [Truststore](#TrustStore) ### Consumer Context Type for the consumer context.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

### CRL File Specifies the location of the certification revocation list (CRL) file.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the CRL file.

### Custom OCSP Responder Configures a custom OCSP responder for certification revocation checks.
FieldTypeDescriptionDefault ValueRequired

Url

String

URL of the OCSP responder.

Cert Alias

String

Alias of the signing certificate for the OCSP response. If specified, the alias must be in the truststore.

### Expiration Policy Configures an expiration policy strategy.
FieldTypeDescriptionDefault ValueRequired

Max Idle Time

Number

Configures the maximum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Max Idle Time field.

### Kafka Message Represents a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Attributes

Attributes associated with the Kafka message.

Payload

Binary

Payload of the Kafka message, represented as an InputStream.

### Kafka Message Attributes Represents the attributes of a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Headers

Object

Headers of the Kafka message, represented as a map in which the key is a string and the value is an InputStream.

Key

Binary

Key of the Kafka message, represented as an InputStream.

Partition

Number

Partition of the topic where the Kafka message is sent.

Topic

String

Topic to which the Kafka message belongs to.

### Kafka Message Metadata Metadata of the Kafka message.
FieldTypeDescriptionDefault ValueRequired

Offset

Number

Offset of the message.

Partition

Number

Partition of the message.

Serialized Key Size

Number

Serialized key size of the message.

Serialized Value Size

Number

Serialized value size of the message.

Timestamp

DateTime

Timestamp of the message.

Topic

String

Topic of the message.

### Kafka Record Attributes Attributes of the Kafka record.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

Creation Timestamp

DateTime

Timestamp of record creation.

Headers

Object

Map of HTTP headers in the message.

Key

Binary

Keys of the record.

Leader Epoch

Number

Leader epoch of the record.

Log Append Timestamp

DateTime

Log append timestamp of the record.

Offset

Number

Offset of the record.

Partition

Number

Partition of the record.

Serialized Key Size

Number

Serialized key size of the record.

Serialized Value Size

Number

Serialized value size of the record.

Topic

String

Topic of the record.

### Keystore Configures the keystore for the TLS protocol. The keystore you generate contains a private key and a public certificate.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the keystore. Mule resolves the path relative to the current classpath and file system.

Type

String

Type of keystore.

Alias

String

Alias of the key to use when the keystore contains multiple private keys. By default, Mule uses the first key in the file.

Key Password

String

Password used to protect the private key.

Password

String

Password used to protect the keystore.

Algorithm

String

Encryption algorithm that the keystore uses.

### Proxy Config Params Configures the proxy configuration parameters.
FieldTypeDescriptionDefault ValueRequired

Host

String

Hostname or IP address of the proxy server.

Port

Number

Port of the proxy server.

Username

String

Username to authenticate against the proxy server.

Password

String

Password to authenticate against the proxy server.

### Reconnect Configures a standard reconnection strategy, which specifies how often to reconnect and how many reconnection attempts the connector source or operation can make.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

Count

Number

How many reconnection attempts the Mule app can make.

### Reconnect Forever Configures a forever reconnection strategy by which the connector source or operation attempts to reconnect at a specified frequency for as long as the Mule app runs.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

### Reconnection Configures a reconnection strategy for an operation.
FieldTypeDescriptionDefault ValueRequired

Fails Deployment

Boolean

What to do if, when an app is deployed, a connectivity test does not pass after exhausting the associated reconnection strategy:

  • true

    Allow the deployment to fail.

  • false

    Ignore the results of the connectivity test.

Reconnection Strategy

Reconnection strategy to use.

### Record Type for records.
FieldTypeDescriptionDefault ValueRequired

Attributes

Kafka record attributes.

Payload

Binary

Payload for the record.

### Redelivery Policy Configures the redelivery policy for executing requests that generate errors. You can add a redelivery policy to any source in a flow.
FieldTypeDescriptionDefault ValueRequired

Max Redelivery Count

Number

Maximum number of times that a redelivered request can be processed unsuccessfully before returning a REDELIVERY_EXHAUSTED error.

Use Secure Hash

Boolean

If true, Mule uses a secure hash algorithm to identify a redelivered message.

Message Digest Algorithm

String

Secure hashing algorithm to use if the Use Secure Hash field is true. If the payload of the message is a Java object, Mule ignores this value and returns the value that the payload’s hashCode() returned.

Id Expression

String

Defines one or more expressions to use to determine when a message has been redelivered. This property may only be set if Use Secure Hash is false.

Object Store

ObjectStore

Configures the object store that stores the redelivery counter for each message.

### Repeatable File Store Stream Configures the repeatable file-store streaming strategy by which Mule keeps a portion of the stream content in memory. If the stream content is larger than the configured buffer size, Mule backs up the buffer’s content to disk and then clears the memory.
FieldTypeDescriptionDefault ValueRequired

In Memory Size

Number

Maximum amount of memory that the stream can use for data. If the amount of memory exceeds this value, Mule buffers the content to disk. To optimize performance:

  • Configure a larger buffer size to avoid the number of times Mule needs to write the buffer on disk. This increases performance, but it also limits the number of concurrent requests your application can process, because it requires additional memory.

  • Configure a smaller buffer size to decrease memory load at the expense of response time.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the In Memory Size field.

### Repeatable In Memory Stream Configures the in-memory streaming strategy by which the request fails if the data exceeds the MAX buffer size. Always run performance tests to find the optimal buffer size for your specific use case.
FieldTypeDescriptionDefault ValueRequired

Initial Buffer Size

Number

Initial amount of memory to allocate to the data stream. If the streamed data exceeds this value, the buffer expands by Buffer Size Increment, with an upper limit of Max In Memory Size value.

Buffer Size Increment

Number

Amount by which the buffer size expands if it exceeds its initial size. Setting a value of 0 or lower specifies that the buffer can’t expand.

Max Buffer Size

Number

Maximum amount of memory to use. If more than that is used then a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value lower than or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the Initial Buffer Size, Buffer Size Increment, and Buffer Unit fields.

### Standard Revocation Check Configures standard revocation checks for TLS certificates.
FieldTypeDescriptionDefault ValueRequired

Only End Entities

Boolean

Which elements to verify in the certificate chain:

  • true

    Verify only the last element in the certificate chain.

  • false

    Verify all elements in the certificate chain.

Prefer Crls

Boolean

How to check certificate validity:

  • true

    Check the Certification Revocation List (CRL) for certificate validity.

  • false

    Use the Online Certificate Status Protocol (OCSP) to check certificate validity.

No Fallback

Boolean

Whether to use the secondary method to check certificate validity:

  • true

    Use the method that wasn’t specified in the Prefer Crls field (the secondary method) to check certificate validity.

  • false

    Do not use the secondary method to check certificate validity.

Soft Fail

Boolean

What to do if the revocation server can’t be reached or is busy:

  • true

    Avoid verification failure.

  • false

    Allow the verification to fail.

### Tcp Client Socket Params Configures the TCP client socket parameters.
FieldTypeDescriptionDefault ValueRequired

Send Buffer Size

Number

Size of the buffer (in bytes) used when sending data, set on the socket itself.

Receive Buffer Size

Number

Buffer size.

Client Timeout

Number

Sets the SO_TIMEOUT value on sockets. Indicates the amount of time (in milliseconds) that the socket waits in a blocking operation before failing. A value of 0 (the default) means waiting indefinitely.

0

Send Tcp No Delay

Boolean

If set, data is sent immediately rather than being collected for efficiency, prioritizing latency over network traffic reduction. Despite the socket default being false, this option defaults to true because reducing network traffic is generally not a major concern.

true

Linger

Number

Sets the SO_LINGER value in milliseconds. This determines how long the socket remains open to ensure all remaining data is transmitted correctly before closing.

Keep Alive

Boolean

Enables the SO_KEEPALIVE behavior on open sockets. This option periodically checks idle socket connections and closes them if they become unavailable. It is used by server sockets to control whether connections are kept alive before being recycled.

false

Connection Timeout

Number

How long the connector waits before timing out when establishing a connection to the remote service.

30000

### TLS Configures TLS to provide secure communications for the Mule app.
FieldTypeDescriptionDefault ValueRequired

Enabled Protocols

String

Comma-separated list of protocols enabled for this context.

Enabled Cipher Suites

String

Comma-separated list of cipher suites enabled for this context.

Trust Store

Configures the TLS truststore.

Key Store

Configures the TLS keystore.

Revocation Check

Configures a revocation checking mechanism.

### Topic Partition Type for topic partition.
FieldTypeDescriptionDefault ValueRequired

Topic

String

Topic name.

x

Partition

Number

Number of partitions.

x

### Truststore Configures the truststore for TLS.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the truststore. Mule resolves the path relative to the current classpath and file system.

Password

String

Password used to protect the truststore.

Type

String

Type of truststore.

Algorithm

String

Encryption algorithm that the truststore uses.

Insecure

Boolean

If true, Mule stops performing certificate validations. Setting this to true can make connections vulnerable to attacks.

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.12 --- # Apache Kafka Connector 4.12 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system and achieve seamless integration between your Mule app and a Kafka cluster, using Mule runtime engine (Mule). For compatibility information, see the Apache Kafka Connector Release Notes. ## Before You Begin Before creating an app, you must: - Have access to the Apache Kafka target resource and Anypoint Platform - Understand how to create a Mule app using Anypoint Studio - Have access to Apache Kafka to get values for the fields that appear in Studio ## Common Use Cases for the Connector Common use cases for Apache Kafka Connector include: - **Log Aggregation** Leverage the low-latency processing from Apache Kafka to collect logs from multiple services and make them available in a standard format to multiple consumers. For example, you can publish a message to Apache Kafka and then retrieve it. - **Analysis and Metrics** Optimize your advertising budget by integrating Apache Kafka and your big data analytics solution to analyze end user activity, such as page views, clicks, shares, and so on, to serve relevant ads. For example, a department store can use a website activity tracker to improve the online shopping experience. The gathered data is sent to multiple departments for various computations. Each department reviews the received information to stay informed about what the customer is looking for and then provides recommendations accordingly. - **Notifications and Alerts** Notify customers about various financial events, such as the recent amount of a transaction or more complex events like future investment suggestions based on integrations with credit agencies, location, and so on. - **Time-Sensitive Applications** Build integration applications that are time-sensitive. For example, you can build an application that ensures that patients in a hospital receive the care they need in a timely manner with a hospital server that processes emergency requests for patient admissions and orders the requests based on priority determined by specified criteria. In this scenario, an application processes Apache Kafka messages in the order that they are received, relying on the order and idempotency of the messages sent through the queue. For example, a newsroom can use the Apache Kafka system to deliver the latest news. Retrieve the latest news by building an integration application by reading from the end of the Apache Kafka queue first. For examples of these use cases, refer to [Apache Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-examples). ### Publish-to Use Cases With Kafka Connector, applications can publish binary messages to the Kafka server. Some examples of use cases include: - Publishing binary data to the Kafka server and the underlying topic. - Setting the batch size of the data to publish to Kafka when working with large volumes of data. - To optimize for performance, companies can batch messages to send. Kafka stores the message until the batch size is reached and completes the send at that time. - Configuring the message size to send to focus on optimal performance. - Kafka is not intended to process and handle large messages. Messages are typically in the 1 MB range and users set the size to avoid performance degradation. In the case of an error, the message is rejected. - Blocking further commits on a configurable level of acknowledgment for the messages published to the Kafka server to ensure successful data transmission. - Acknowledgments enable users to ensure that the message is sent prior to publishing more messages. This is specifically useful in time-ordered or event-ordered messages, such as logs or patient drug administration orders. Ensuring that messages are sent in order helps the downstream application process the messages in the right order. - Configuring how data is serialized during a publish. - Securing messages sent during Mutual TLS. The following lists the acceptance criteria for publishing: - Publish a message to a Kafka topic. - Configure the batch size and size of the message. - Obtain an acknowledgment once publishing is complete at the configured level. - Send the message securely via Mutual TLS. - Configure the serialization type. ### Subscribe-from Use Cases With Kafka Connector, applications can obtain messages from Kafka for further consumption downstream. Some examples of use cases include: Reading from Kafka: - Subscribing to the Kafka server and the underlying group to read and consume messages for downstream processing. - Subscribing to a specific Kafka partition to process messages. - Users with a high availability microservice don’t want to rely on Kafka’s rebalancing in the event of a failure. - If users have a local state associated with their partition, they want to process only that partition. - Configuring the offset value to a set value or to the beginning or end of the topic. - Users have a local store and want to use that information to start processing their records. - Users have time-sensitive records and want to skip to the end if the application must be up-to-date or skip to the beginning if it is a new application processing the records. - Reading the offset value from an alternate storage system on restart. - Users have a local store with the offset information and want to read the value from their local storage. - Setting the interval for which to block the receipt of messages for varying downstream processing intervals. Processing the messages: - Ensuring there is no data loss when the downstream application crashes by manually configuring commits to have more control. - Storing a commit value in a local store or in the Kafka storage and blocking until the store is successful. - Storing an offset asynchronously in a local store or in the Kafka storage. - Using Kafka’s in-built capability of load-balancing with multiple applications reading from the same group ID. - Configuring the number of records to process in a single call. The following lists the acceptance criteria for subscribing: - Read a message from Kafka from a specific partition or topic. - Read a message from Kafka after obtaining the offset from a local store. - Read a message from the beginning of the queue, the end of the queue, and a pre-specified offset. - Ensure a high level of performance when there is one consumer per thread. Multiple consumers exist in a single group. ## Audience - Starting user To create a Mule app, see [Apache Kafka Studio Configuration](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-studio). - Power user Read [XML and Maven Support](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-xml-maven) and [Examples](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-examples). ## Next After you complete the prerequisites, you are ready to create an app with [Anypoint Studio](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-studio). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Use Exchange to Discover Connectors, Templates, and Examples](https://docs.mulesoft.com/connectors/introduction/intro-use-exchange) - [Apache Kafka Connector](https://www.mulesoft.com/exchange/com.mulesoft.connectors/mule-kafka-connector/) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.12 --- # Apache Kafka Connector 4.12 - Additional Configuration Information > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Set additional configuration for Anypoint Connector for Apache Kafka (Apache Kafka Connector). ## Configure a Custom Partitioner You can configure a custom partitioner for Apache Kafka Connector in a Mule app: 1. Use Maven to create a Java project and add the custom partitioner to it: ```java public class CustomPartitioner implements Partitioner { // Implementation } ``` 2. Add this dependency in the Mule app: ```java org.example custom-partitioner 1.0-SNAPSHOT ``` 3. Add this dependency to `additionalPluginDependencies`: ```java org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 org.example custom-partitioner 1.0-SNAPSHOT mule-application ``` 4. Add the custom partitioner to the producer configuration: ```java ``` ## Next Step After you complete configuring the connector, you can try the [examples](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.12 --- # Apache Kafka Connector 4.12 Reference > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system. It provides seamless integration between your Mule app and an Apache Kafka cluster, using Mule runtime engine (Mule). > [!NOTE] When a **Message Listener** or a **Batch Message Listener** uses the `MANUAL` > acknowledgement mode, you must use a **Commit** operation at the end of the flow when the flow > finishes successfully (either because there were no errors or because there is an **On Error > Continue** component in the flow). > [!NOTE] Mule server notifications are not supported because the Kafka library used in the > connector manages network disruptions internally. ## Consumer Configuration Configuration for consumers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default acknowledgement mode

Enumeration, one of

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

AUTO

Default listener poll timeout

Number

Time to wait to perform a poll if data is not available in the buffer (fetched). If no value is set, the poll is returned immediately with any records that are currently available in the buffer or else returns empty if there is no data. Must not be negative.

100

Default listener poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default listener poll timeout field. This combines with Poll Timeout to define the total timeout for the polling.

MILLISECONDS

Default operation poll timeout

Number

Time to wait for an operation to finish. If no value is set or a negative value is set, the operation waits forever.

-1

Default operation poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default operation timeout field. This combines with Operation Timeout to define the total default timeout for the operations that use this configuration.

SECONDS

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Consumer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. Additionally, the connector supports only [RFC6749 standard](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4.3). For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Group ID

String

Default Group ID for all the Kafka Consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Whether internal topics matching a subscribed pattern is excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the reconnect backoff timeout scalar.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

true

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this field at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. You can override this field at the source level.

KB

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this field at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Request Timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This parameter can be overridden at source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Default record limit

Number

The maximum number of records returned on a poll call to the Kafka cluster. This parameter can be overridden at source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

The expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch heartbeat interval time scalar.

SECONDS

Session timeout

Number

The timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for session timeout scalar.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this config.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for connections maximum idle time scalar.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Topic Subscription Patterns

Array of String

The list of subscription regular expressions to subscribe to. This topics will be automatically rebalanced between the amount of consumers of the topic.

Assignments

The list of topic-partition pairs to assign. Note that there will be no automatic rebalance of the consumers

Default fetch minimum size

Number

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request waits for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency. This parameter can be overridden at source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

BYTE

Default fetch maximum size

Number

The maximum amount of data the server should return for a fetch request. Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). Note that the consumer performs multiple fetches in parallel. This parameter can be overridden at source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Default maximum partition fetch size

Number

The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size.This parameter can be overridden at source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Fetch Maximum Wait Timeout

Number

The maximum amount of time the server will block before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch maximum wait timeout scalar.

MILLISECONDS

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Consumer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Encryption type

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation. Enable FIPS mode by setting the `com.mulesoft.connectors.allow-registration-of-sasl-security-provider-in-fips=true` flag to true. This adds the `com.sun.security.sasl.Provider` security provider in the FIPS environment.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

User used by the client to connect to the Kafka broker.

x

Password

String

Password used by the client to connect to the Kafka broker.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

--- ## Producer Configuration Configuration for producers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default topic

String

Default topic name to use by the producer operations, overridable at the operation’s configuration level.

defaultTopicName

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Producer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names..

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Batch size

Number

The producer attempts to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size. Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent. A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the batch size scalar.

KB

Buffer size

Number

The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block for max.block.ms after which it will throw an exception. This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests. The default value in the Kafka docs is of 33554432 (32MB), but this should be capped to align with expected values for mule instances in cloudhub (v0.1 core)

1000

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

KB

DNS lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).

NONE

Connections maximum idle time

Number

Close idle connections after the value specified by this config.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the connections maximum idle scalar.

SECONDS

Delivery timeout

Number

An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retriable send failures. The producer may report failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch which reached an earlier delivery expiration deadline. The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the delivery timeout scalar.

SECONDS

Enable idempotence

Boolean

When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc, may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0 and acks must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConnectionException will be thrown

false

Linger time

Number

The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay?that is, rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it is sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the linger time scalar.

SECONDS

Maximum block time

Number

The configuration controls how long KafkaProducer.send() and KafkaProducer.partitionsFor() will block.These methods can be blocked either because the buffer is full or metadata unavailable.Blocking in the user-supplied serializers or partitioner will not be counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the maximum block time scalar.

SECONDS

Maximum in flight requests

Number

The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).

5

Maximum request size

Number

The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size which may be different from this.

1

Maximum request size unit.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

MB

Producer acknowledge mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent

NONE

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. This parameter can be overridden at source level.

KB

Retries amount

Number

Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first. Note additionally that produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires first before successful acknowledgment. Users should generally prefer to leave this config unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the retry backoff timeout time scalar.

MILLISECONDS

Retry backoff timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the request timeout time scalar.

SECONDS

Default request timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This should be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

The configuration controls which partitioning strategy is used when sending messages without providing a key or a partition.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Producer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

EncryptionType

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

## Consumer Sources - [Batch Message Listener](#batch-message-listener) - [Message Listener](#message-listener) ### Batch Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a list of messages to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Record

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) > [!NOTE] The **Batch Message Listener** source does not support configurable streaming strategies > because the source takes a batch of records as an input stream. The streaming strategy > configuration is non-repeatable-stream by default for the source. ### Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a single message to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) ## Consumer Operations - [Commit](#commit) - [Consume](#consume) - [Seek](#seek) ### Commit `` Commits the offsets associated to a message or batch of messages consumed in a **Message Listener**. This is a list or a single message consumed in the **Batch Message Listener** source.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumer commit key

String

Commit key of the last poll. This operation is valid only when used inside a flow that uses one of the sources (Batch Message Listener or Message Listener) which inserts this value as an attribute in the Mule event.

x

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_ACK\_MODE - KAFKA:RETRY\_EXHAUSTED - KAFKA:ALREADY\_COMMITTED - KAFKA:TIMEOUT - KAFKA:SESSION\_NOT\_FOUND - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Consume `` Receives messages from one or more Kafka topics. This operation works similarly to the **Message Listener** source, so all the operations that apply to that, apply to this operation as well. > [!NOTE] The **Consume** operation does not return the `consumerCommitKey`.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumption timeout

Number

Number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Consumption timeout field.

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

IMMEDIATE

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:SECURITY - KAFKA:RETRY\_EXHAUSTED - KAFKA:ILLEGAL\_STATE - KAFKA:TIMEOUT - KAFKA:SERIALIZATION\_ERROR - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Seek `` Sets the current offset of the consumer for the given topic and partition to the provided offset value.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which the Seek operation is performed.

x

Partition

Number

Partition number that has its offset modified.

x

Offset

Number

Offset value to commit for the configured partition.

x

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:INVALID\_TOPIC - KAFKA:SECURITY - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ## Producer Operations - [Bulk Publish](#bulkPublish) - [Publish](#publish) ### Bulk Publish `` Publishes a bulk of messages to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Messages

Array of Kafka Message

List of messages to publish.

#[payload]

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:AUTHENTICATION\_ERROR - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ### Publish `` Publishes a message to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Topic to publish to.

Partition

Number

(Optional) Topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ## Object Types - [Consumer Context](#ConsumerContext) - [CRL File](#CrlFile) - [Custom OCSP Responder](#CustomOcspResponder) - [Expiration Policy](#ExpirationPolicy) - [Kafka Message](#KafkaMessage) - [Kafka Message Attributes](#KafkaMessageAttributes) - [Kafka Message Metadata](#KafkaMessageMetadata) - [Kafka Record Attributes](#KafkaRecordAttributes) - [Keystore](#KeyStore) - [Proxy Config Params](#ProxyConfigParams) - [Reconnect](#Reconnect) - [Reconnect Forever](#Reconnect-Forever) - [Reconnection](#Reconnection) - [Record](#Record) - [Redelivery Policy](#RedeliveryPolicy) - [Repeatable File Store Stream](#RepeatableFileStoreStream) - [Repeatable In Memory Stream](#RepeatableInMemoryStream) - [Standard Revocation Check](#StandardRevocationCheck) - [Tcp Client Socket Params](#TcpClientSocketParams) - [TLS](#Tls) - [Topic Partition](#TopicPartition) - [Truststore](#TrustStore) ### Consumer Context Type for the consumer context.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

### CRL File Specifies the location of the certification revocation list (CRL) file.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the CRL file.

### Custom OCSP Responder Configures a custom OCSP responder for certification revocation checks.
FieldTypeDescriptionDefault ValueRequired

Url

String

URL of the OCSP responder.

Cert Alias

String

Alias of the signing certificate for the OCSP response. If specified, the alias must be in the truststore.

### Expiration Policy Configures an expiration policy strategy.
FieldTypeDescriptionDefault ValueRequired

Max Idle Time

Number

Configures the maximum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Max Idle Time field.

### Kafka Message Represents a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Attributes

Attributes associated with the Kafka message.

Payload

Binary

Payload of the Kafka message, represented as an InputStream.

### Kafka Message Attributes Represents the attributes of a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Headers

Object

Headers of the Kafka message, represented as a map in which the key is a string and the value is an InputStream.

Key

Binary

Key of the Kafka message, represented as an InputStream.

Partition

Number

Partition of the topic where the Kafka message is sent.

Topic

String

Topic to which the Kafka message belongs to.

### Kafka Message Metadata Metadata of the Kafka message.
FieldTypeDescriptionDefault ValueRequired

Offset

Number

Offset of the message.

Partition

Number

Partition of the message.

Serialized Key Size

Number

Serialized key size of the message.

Serialized Value Size

Number

Serialized value size of the message.

Timestamp

DateTime

Timestamp of the message.

Topic

String

Topic of the message.

### Kafka Record Attributes Attributes of the Kafka record.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

Creation Timestamp

DateTime

Timestamp of record creation.

Headers

Object

Map of HTTP headers in the message.

Key

Binary

Keys of the record.

Leader Epoch

Number

Leader epoch of the record.

Log Append Timestamp

DateTime

Log append timestamp of the record.

Offset

Number

Offset of the record.

Partition

Number

Partition of the record.

Serialized Key Size

Number

Serialized key size of the record.

Serialized Value Size

Number

Serialized value size of the record.

Topic

String

Topic of the record.

### Keystore Configures the keystore for the TLS protocol. The keystore you generate contains a private key and a public certificate.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the keystore. Mule resolves the path relative to the current classpath and file system.

Type

String

Type of keystore.

Alias

String

Alias of the key to use when the keystore contains multiple private keys. By default, Mule uses the first key in the file.

Key Password

String

Password used to protect the private key.

Password

String

Password used to protect the keystore.

Algorithm

String

Encryption algorithm that the keystore uses.

### Proxy Config Params Configures the proxy configuration parameters.
FieldTypeDescriptionDefault ValueRequired

Host

String

Hostname or IP address of the proxy server.

Port

Number

Port of the proxy server.

Username

String

Username to authenticate against the proxy server.

Password

String

Password to authenticate against the proxy server.

### Reconnect Configures a standard reconnection strategy, which specifies how often to reconnect and how many reconnection attempts the connector source or operation can make.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

Count

Number

How many reconnection attempts the Mule app can make.

### Reconnect Forever Configures a forever reconnection strategy by which the connector source or operation attempts to reconnect at a specified frequency for as long as the Mule app runs.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

### Reconnection Configures a reconnection strategy for an operation.
FieldTypeDescriptionDefault ValueRequired

Fails Deployment

Boolean

What to do if, when an app is deployed, a connectivity test does not pass after exhausting the associated reconnection strategy:

  • true

    Allow the deployment to fail.

  • false

    Ignore the results of the connectivity test.

Reconnection Strategy

Reconnection strategy to use.

### Record Type for records.
FieldTypeDescriptionDefault ValueRequired

Attributes

Kafka record attributes.

Payload

Binary

Payload for the record.

### Redelivery Policy Configures the redelivery policy for executing requests that generate errors. You can add a redelivery policy to any source in a flow.
FieldTypeDescriptionDefault ValueRequired

Max Redelivery Count

Number

Maximum number of times that a redelivered request can be processed unsuccessfully before returning a REDELIVERY_EXHAUSTED error.

Use Secure Hash

Boolean

If true, Mule uses a secure hash algorithm to identify a redelivered message.

Message Digest Algorithm

String

Secure hashing algorithm to use if the Use Secure Hash field is true. If the payload of the message is a Java object, Mule ignores this value and returns the value that the payload’s hashCode() returned.

Id Expression

String

Defines one or more expressions to use to determine when a message has been redelivered. This property may only be set if Use Secure Hash is false.

Object Store

ObjectStore

Configures the object store that stores the redelivery counter for each message.

### Repeatable File Store Stream Configures the repeatable file-store streaming strategy by which Mule keeps a portion of the stream content in memory. If the stream content is larger than the configured buffer size, Mule backs up the buffer’s content to disk and then clears the memory.
FieldTypeDescriptionDefault ValueRequired

In Memory Size

Number

Maximum amount of memory that the stream can use for data. If the amount of memory exceeds this value, Mule buffers the content to disk. To optimize performance:

  • Configure a larger buffer size to avoid the number of times Mule needs to write the buffer on disk. This increases performance, but it also limits the number of concurrent requests your application can process, because it requires additional memory.

  • Configure a smaller buffer size to decrease memory load at the expense of response time.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the In Memory Size field.

### Repeatable In Memory Stream Configures the in-memory streaming strategy by which the request fails if the data exceeds the MAX buffer size. Always run performance tests to find the optimal buffer size for your specific use case.
FieldTypeDescriptionDefault ValueRequired

Initial Buffer Size

Number

Initial amount of memory to allocate to the data stream. If the streamed data exceeds this value, the buffer expands by Buffer Size Increment, with an upper limit of Max In Memory Size value.

Buffer Size Increment

Number

Amount by which the buffer size expands if it exceeds its initial size. Setting a value of 0 or lower specifies that the buffer can’t expand.

Max Buffer Size

Number

Maximum amount of memory to use. If more than that is used then a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value lower than or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the Initial Buffer Size, Buffer Size Increment, and Buffer Unit fields.

### Standard Revocation Check Configures standard revocation checks for TLS certificates.
FieldTypeDescriptionDefault ValueRequired

Only End Entities

Boolean

Which elements to verify in the certificate chain:

  • true

    Verify only the last element in the certificate chain.

  • false

    Verify all elements in the certificate chain.

Prefer Crls

Boolean

How to check certificate validity:

  • true

    Check the Certification Revocation List (CRL) for certificate validity.

  • false

    Use the Online Certificate Status Protocol (OCSP) to check certificate validity.

No Fallback

Boolean

Whether to use the secondary method to check certificate validity:

  • true

    Use the method that wasn’t specified in the Prefer Crls field (the secondary method) to check certificate validity.

  • false

    Do not use the secondary method to check certificate validity.

Soft Fail

Boolean

What to do if the revocation server can’t be reached or is busy:

  • true

    Avoid verification failure.

  • false

    Allow the verification to fail.

### Tcp Client Socket Params Configures the TCP client socket parameters.
FieldTypeDescriptionDefault ValueRequired

Send Buffer Size

Number

Size of the buffer (in bytes) used when sending data, set on the socket itself.

Receive Buffer Size

Number

Buffer size.

Client Timeout

Number

Sets the SO_TIMEOUT value on sockets. Indicates the amount of time (in milliseconds) that the socket waits in a blocking operation before failing. A value of 0 (the default) means waiting indefinitely.

0

Send Tcp No Delay

Boolean

If set, data is sent immediately rather than being collected for efficiency, prioritizing latency over network traffic reduction. Despite the socket default being false, this option defaults to true because reducing network traffic is generally not a major concern.

true

Linger

Number

Sets the SO_LINGER value in milliseconds. This determines how long the socket remains open to ensure all remaining data is transmitted correctly before closing.

Keep Alive

Boolean

Enables the SO_KEEPALIVE behavior on open sockets. This option periodically checks idle socket connections and closes them if they become unavailable. It is used by server sockets to control whether connections are kept alive before being recycled.

false

Connection Timeout

Number

How long the connector waits before timing out when establishing a connection to the remote service.

30000

### TLS Configures TLS to provide secure communications for the Mule app.
FieldTypeDescriptionDefault ValueRequired

Enabled Protocols

String

Comma-separated list of protocols enabled for this context.

Enabled Cipher Suites

String

Comma-separated list of cipher suites enabled for this context.

Trust Store

Configures the TLS truststore.

Key Store

Configures the TLS keystore.

Revocation Check

Configures a revocation checking mechanism.

### Topic Partition Type for topic partition.
FieldTypeDescriptionDefault ValueRequired

Topic

String

Topic name.

x

Partition

Number

Number of partitions.

x

### Truststore Configures the truststore for TLS.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the truststore. Mule resolves the path relative to the current classpath and file system.

Password

String

Password used to protect the truststore.

Type

String

Type of truststore.

Algorithm

String

Encryption algorithm that the truststore uses.

Insecure

Boolean

If true, Mule stops performing certificate validations. Setting this to true can make connections vulnerable to attacks.

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.13 --- # Apache Kafka Connector 4.13 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system and achieve seamless integration between your Mule app and a Kafka cluster, using Mule runtime engine (Mule). For compatibility information, see the Apache Kafka Connector Release Notes. ## Before You Begin Before creating an app, you must: - Have access to the Apache Kafka target resource and Anypoint Platform - Understand how to create a Mule app using Anypoint Studio - Have access to Apache Kafka to get values for the fields that appear in Studio ## Common Use Cases for the Connector Common use cases for Apache Kafka Connector include: - **Log Aggregation** Leverage the low-latency processing from Apache Kafka to collect logs from multiple services and make them available in a standard format to multiple consumers. For example, you can publish a message to Apache Kafka and then retrieve it. - **Analysis and Metrics** Optimize your advertising budget by integrating Apache Kafka and your big data analytics solution to analyze end user activity, such as page views, clicks, shares, and so on, to serve relevant ads. For example, a department store can use a website activity tracker to improve the online shopping experience. The gathered data is sent to multiple departments for various computations. Each department reviews the received information to stay informed about what the customer is looking for and then provides recommendations accordingly. - **Notifications and Alerts** Notify customers about various financial events, such as the recent amount of a transaction or more complex events like future investment suggestions based on integrations with credit agencies, location, and so on. - **Time-Sensitive Applications** Build integration applications that are time-sensitive. For example, you can build an application that ensures that patients in a hospital receive the care they need in a timely manner with a hospital server that processes emergency requests for patient admissions and orders the requests based on priority determined by specified criteria. In this scenario, an application processes Apache Kafka messages in the order that they are received, relying on the order and idempotency of the messages sent through the queue. For example, a newsroom can use the Apache Kafka system to deliver the latest news. Retrieve the latest news by building an integration application by reading from the end of the Apache Kafka queue first. For examples of these use cases, refer to [Apache Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-examples). ### Publish-to Use Cases With Kafka Connector, applications can publish binary messages to the Kafka server. Some examples of use cases include: - Publishing binary data to the Kafka server and the underlying topic. - Setting the batch size of the data to publish to Kafka when working with large volumes of data. - To optimize for performance, companies can batch messages to send. Kafka stores the message until the batch size is reached and completes the send at that time. - Configuring the message size to send to focus on optimal performance. - Kafka is not intended to process and handle large messages. Messages are typically in the 1 MB range and users set the size to avoid performance degradation. In the case of an error, the message is rejected. - Blocking further commits on a configurable level of acknowledgment for the messages published to the Kafka server to ensure successful data transmission. - Acknowledgments enable users to ensure that the message is sent prior to publishing more messages. This is specifically useful in time-ordered or event-ordered messages, such as logs or patient drug administration orders. Ensuring that messages are sent in order helps the downstream application process the messages in the right order. - Configuring how data is serialized during a publish. - Securing messages sent during Mutual TLS. The following lists the acceptance criteria for publishing: - Publish a message to a Kafka topic. - Configure the batch size and size of the message. - Obtain an acknowledgment once publishing is complete at the configured level. - Send the message securely via Mutual TLS. - Configure the serialization type. ### Subscribe-from Use Cases With Kafka Connector, applications can obtain messages from Kafka for further consumption downstream. Some examples of use cases include: Reading from Kafka: - Subscribing to the Kafka server and the underlying group to read and consume messages for downstream processing. - Subscribing to a specific Kafka partition to process messages. - Users with a high availability microservice don’t want to rely on Kafka’s rebalancing in the event of a failure. - If users have a local state associated with their partition, they want to process only that partition. - Configuring the offset value to a set value or to the beginning or end of the topic. - Users have a local store and want to use that information to start processing their records. - Users have time-sensitive records and want to skip to the end if the application must be up-to-date or skip to the beginning if it is a new application processing the records. - Reading the offset value from an alternate storage system on restart. - Users have a local store with the offset information and want to read the value from their local storage. - Setting the interval for which to block the receipt of messages for varying downstream processing intervals. Processing the messages: - Ensuring there is no data loss when the downstream application crashes by manually configuring commits to have more control. - Storing a commit value in a local store or in the Kafka storage and blocking until the store is successful. - Storing an offset asynchronously in a local store or in the Kafka storage. - Using Kafka’s in-built capability of load-balancing with multiple applications reading from the same group ID. - Configuring the number of records to process in a single call. The following lists the acceptance criteria for subscribing: - Read a message from Kafka from a specific partition or topic. - Read a message from Kafka after obtaining the offset from a local store. - Read a message from the beginning of the queue, the end of the queue, and a pre-specified offset. - Ensure a high level of performance when there is one consumer per thread. Multiple consumers exist in a single group. ## Audience - Starting user To create a Mule app, see [Apache Kafka Studio Configuration](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-studio). - Power user Read [XML and Maven Support](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-xml-maven) and [Examples](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-examples). ## Next After you complete the prerequisites, you are ready to create an app with [Anypoint Studio](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-studio). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Use Exchange to Discover Connectors, Templates, and Examples](https://docs.mulesoft.com/connectors/introduction/intro-use-exchange) - [Apache Kafka Connector](https://www.mulesoft.com/exchange/com.mulesoft.connectors/mule-kafka-connector/) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.13 --- # Apache Kafka Connector 4.13 - Additional Configuration Information > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Set additional configuration for Anypoint Connector for Apache Kafka (Apache Kafka Connector). ## Configure a Custom Partitioner You can configure a custom partitioner for Apache Kafka Connector in a Mule app: 1. Use Maven to create a Java project and add the custom partitioner to it: ```java public class CustomPartitioner implements Partitioner { // Implementation } ``` 2. Add this dependency in the Mule app: ```java org.example custom-partitioner 1.0-SNAPSHOT ``` 3. Add this dependency to `additionalPluginDependencies`: ```java org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 org.example custom-partitioner 1.0-SNAPSHOT mule-application ``` 4. Add the custom partitioner to the producer configuration: ```java ``` ## Next Step After you complete configuring the connector, you can try the [examples](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.13 --- # Apache Kafka Connector 4.13 Reference > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system. It provides seamless integration between your Mule app and an Apache Kafka cluster, using Mule runtime engine (Mule). > [!NOTE] When a **Message Listener** or a **Batch Message Listener** uses the `MANUAL` > acknowledgement mode, you must use a **Commit** operation at the end of the flow when the flow > finishes successfully (either because there were no errors or because there is an **On Error > Continue** component in the flow). > [!NOTE] Mule server notifications are not supported because the Kafka library used in the > connector manages network disruptions internally. ## Consumer Configuration Configuration for consumers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default acknowledgement mode

Enumeration, one of

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

AUTO

Default listener poll timeout

Number

Time to wait to perform a poll if data is not available in the buffer (fetched). If no value is set, the poll is returned immediately with any records that are currently available in the buffer or else returns empty if there is no data. Must not be negative.

100

Default listener poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default listener poll timeout field. This combines with Poll Timeout to define the total timeout for the polling.

MILLISECONDS

Default operation poll timeout

Number

Time to wait for an operation to finish. If no value is set or a negative value is set, the operation waits forever.

-1

Default operation poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default operation timeout field. This combines with Operation Timeout to define the total default timeout for the operations that use this configuration.

SECONDS

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Consumer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. Additionally, the connector supports only [RFC6749 standard](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4.3). For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Group ID

String

Default Group ID for all the Kafka Consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Whether internal topics matching a subscribed pattern is excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the reconnect backoff timeout scalar.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

true

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this field at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. You can override this field at the source level.

KB

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this field at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Request Timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This parameter can be overridden at source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Default record limit

Number

The maximum number of records returned on a poll call to the Kafka cluster. This parameter can be overridden at source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

The expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch heartbeat interval time scalar.

SECONDS

Session timeout

Number

The timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for session timeout scalar.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this config.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for connections maximum idle time scalar.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Topic Subscription Patterns

Array of String

The list of subscription regular expressions to subscribe to. This topics will be automatically rebalanced between the amount of consumers of the topic.

Assignments

The list of topic-partition pairs to assign. Note that there will be no automatic rebalance of the consumers

Default fetch minimum size

Number

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request waits for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency. This parameter can be overridden at source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

BYTE

Default fetch maximum size

Number

The maximum amount of data the server should return for a fetch request. Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). Note that the consumer performs multiple fetches in parallel. This parameter can be overridden at source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Default maximum partition fetch size

Number

The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size.This parameter can be overridden at source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Fetch Maximum Wait Timeout

Number

The maximum amount of time the server will block before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch maximum wait timeout scalar.

MILLISECONDS

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Consumer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Encryption type

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation. Enable FIPS mode by setting the `com.mulesoft.connectors.allow-registration-of-sasl-security-provider-in-fips=true` flag to true. This adds the `com.sun.security.sasl.Provider` security provider in the FIPS environment.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

User used by the client to connect to the Kafka broker.

x

Password

String

Password used by the client to connect to the Kafka broker.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

--- ## Producer Configuration Configuration for producers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default topic

String

Default topic name to use by the producer operations, overridable at the operation’s configuration level.

defaultTopicName

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Producer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names..

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Batch size

Number

The producer attempts to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size. Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent. A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the batch size scalar.

KB

Buffer size

Number

The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block for max.block.ms after which it will throw an exception. This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests. The default value in the Kafka docs is of 33554432 (32MB), but this should be capped to align with expected values for mule instances in cloudhub (v0.1 core)

1000

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

KB

DNS lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).

NONE

Connections maximum idle time

Number

Close idle connections after the value specified by this config.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the connections maximum idle scalar.

SECONDS

Delivery timeout

Number

An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retriable send failures. The producer may report failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch which reached an earlier delivery expiration deadline. The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the delivery timeout scalar.

SECONDS

Enable idempotence

Boolean

When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc, may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0 and acks must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConnectionException will be thrown

false

Linger time

Number

The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay?that is, rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it is sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the linger time scalar.

SECONDS

Maximum block time

Number

The configuration controls how long KafkaProducer.send() and KafkaProducer.partitionsFor() will block.These methods can be blocked either because the buffer is full or metadata unavailable.Blocking in the user-supplied serializers or partitioner will not be counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the maximum block time scalar.

SECONDS

Maximum in flight requests

Number

The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).

5

Maximum request size

Number

The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size which may be different from this.

1

Maximum request size unit.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

MB

Producer acknowledge mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent

NONE

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. This parameter can be overridden at source level.

KB

Retries amount

Number

Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first. Note additionally that produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires first before successful acknowledgment. Users should generally prefer to leave this config unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the retry backoff timeout time scalar.

MILLISECONDS

Retry backoff timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the request timeout time scalar.

SECONDS

Default request timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This should be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

The configuration controls which partitioning strategy is used when sending messages without providing a key or a partition.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Producer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

EncryptionType

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

## Consumer Sources - [Batch Message Listener](#batch-message-listener) - [Message Listener](#message-listener) ### Batch Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a list of messages to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Record

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) > [!NOTE] The **Batch Message Listener** source does not support configurable streaming strategies > because the source takes a batch of records as an input stream. The streaming strategy > configuration is non-repeatable-stream by default for the source. ### Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a single message to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) ## Consumer Operations - [Commit](#commit) - [Consume](#consume) - [Seek](#seek) ### Commit `` Commits the offsets associated to a message or batch of messages consumed in a **Message Listener**. This is a list or a single message consumed in the **Batch Message Listener** source.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumer commit key

String

Commit key of the last poll. This operation is valid only when used inside a flow that uses one of the sources (Batch Message Listener or Message Listener) which inserts this value as an attribute in the Mule event.

x

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_ACK\_MODE - KAFKA:RETRY\_EXHAUSTED - KAFKA:ALREADY\_COMMITTED - KAFKA:TIMEOUT - KAFKA:SESSION\_NOT\_FOUND - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Consume `` Receives messages from one or more Kafka topics. This operation works similarly to the **Message Listener** source, so all the operations that apply to that, apply to this operation as well. > [!NOTE] The **Consume** operation does not return the `consumerCommitKey`.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumption timeout

Number

Number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Consumption timeout field.

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

IMMEDIATE

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:SECURITY - KAFKA:RETRY\_EXHAUSTED - KAFKA:ILLEGAL\_STATE - KAFKA:TIMEOUT - KAFKA:SERIALIZATION\_ERROR - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Seek `` Sets the current offset of the consumer for the given topic and partition to the provided offset value.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which the Seek operation is performed.

x

Partition

Number

Partition number that has its offset modified.

x

Offset

Number

Offset value to commit for the configured partition.

x

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:INVALID\_TOPIC - KAFKA:SECURITY - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ## Producer Operations - [Bulk Publish](#bulkPublish) - [Publish](#publish) ### Bulk Publish `` Publishes a bulk of messages to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Messages

Array of Kafka Message

List of messages to publish.

#[payload]

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:AUTHENTICATION\_ERROR - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ### Publish `` Publishes a message to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Topic to publish to.

Partition

Number

(Optional) Topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ## Object Types - [Consumer Context](#ConsumerContext) - [CRL File](#CrlFile) - [Custom OCSP Responder](#CustomOcspResponder) - [Expiration Policy](#ExpirationPolicy) - [Kafka Message](#KafkaMessage) - [Kafka Message Attributes](#KafkaMessageAttributes) - [Kafka Message Metadata](#KafkaMessageMetadata) - [Kafka Record Attributes](#KafkaRecordAttributes) - [Keystore](#KeyStore) - [Proxy Config Params](#ProxyConfigParams) - [Reconnect](#Reconnect) - [Reconnect Forever](#Reconnect-Forever) - [Reconnection](#Reconnection) - [Record](#Record) - [Redelivery Policy](#RedeliveryPolicy) - [Repeatable File Store Stream](#RepeatableFileStoreStream) - [Repeatable In Memory Stream](#RepeatableInMemoryStream) - [Standard Revocation Check](#StandardRevocationCheck) - [Tcp Client Socket Params](#TcpClientSocketParams) - [TLS](#Tls) - [Topic Partition](#TopicPartition) - [Truststore](#TrustStore) ### Consumer Context Type for the consumer context.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

### CRL File Specifies the location of the certification revocation list (CRL) file.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the CRL file.

### Custom OCSP Responder Configures a custom OCSP responder for certification revocation checks.
FieldTypeDescriptionDefault ValueRequired

Url

String

URL of the OCSP responder.

Cert Alias

String

Alias of the signing certificate for the OCSP response. If specified, the alias must be in the truststore.

### Expiration Policy Configures an expiration policy strategy.
FieldTypeDescriptionDefault ValueRequired

Max Idle Time

Number

Configures the maximum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Max Idle Time field.

### Kafka Message Represents a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Attributes

Attributes associated with the Kafka message.

Payload

Binary

Payload of the Kafka message, represented as an InputStream.

### Kafka Message Attributes Represents the attributes of a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Headers

Object

Headers of the Kafka message, represented as a map in which the key is a string and the value is an InputStream.

Key

Binary

Key of the Kafka message, represented as an InputStream.

Partition

Number

Partition of the topic where the Kafka message is sent.

Topic

String

Topic to which the Kafka message belongs to.

### Kafka Message Metadata Metadata of the Kafka message.
FieldTypeDescriptionDefault ValueRequired

Offset

Number

Offset of the message.

Partition

Number

Partition of the message.

Serialized Key Size

Number

Serialized key size of the message.

Serialized Value Size

Number

Serialized value size of the message.

Timestamp

DateTime

Timestamp of the message.

Topic

String

Topic of the message.

### Kafka Record Attributes Attributes of the Kafka record.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

Creation Timestamp

DateTime

Timestamp of record creation.

Headers

Object

Map of HTTP headers in the message.

Key

Binary

Keys of the record.

Leader Epoch

Number

Leader epoch of the record.

Log Append Timestamp

DateTime

Log append timestamp of the record.

Offset

Number

Offset of the record.

Partition

Number

Partition of the record.

Serialized Key Size

Number

Serialized key size of the record.

Serialized Value Size

Number

Serialized value size of the record.

Topic

String

Topic of the record.

### Keystore Configures the keystore for the TLS protocol. The keystore you generate contains a private key and a public certificate.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the keystore. Mule resolves the path relative to the current classpath and file system.

Type

String

Type of keystore.

Alias

String

Alias of the key to use when the keystore contains multiple private keys. By default, Mule uses the first key in the file.

Key Password

String

Password used to protect the private key.

Password

String

Password used to protect the keystore.

Algorithm

String

Encryption algorithm that the keystore uses.

### Proxy Config Params Configures the proxy configuration parameters.
FieldTypeDescriptionDefault ValueRequired

Host

String

Hostname or IP address of the proxy server.

Port

Number

Port of the proxy server.

Username

String

Username to authenticate against the proxy server.

Password

String

Password to authenticate against the proxy server.

### Reconnect Configures a standard reconnection strategy, which specifies how often to reconnect and how many reconnection attempts the connector source or operation can make.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

Count

Number

How many reconnection attempts the Mule app can make.

### Reconnect Forever Configures a forever reconnection strategy by which the connector source or operation attempts to reconnect at a specified frequency for as long as the Mule app runs.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

### Reconnection Configures a reconnection strategy for an operation.
FieldTypeDescriptionDefault ValueRequired

Fails Deployment

Boolean

What to do if, when an app is deployed, a connectivity test does not pass after exhausting the associated reconnection strategy:

  • true

    Allow the deployment to fail.

  • false

    Ignore the results of the connectivity test.

Reconnection Strategy

Reconnection strategy to use.

### Record Type for records.
FieldTypeDescriptionDefault ValueRequired

Attributes

Kafka record attributes.

Payload

Binary

Payload for the record.

### Redelivery Policy Configures the redelivery policy for executing requests that generate errors. You can add a redelivery policy to any source in a flow.
FieldTypeDescriptionDefault ValueRequired

Max Redelivery Count

Number

Maximum number of times that a redelivered request can be processed unsuccessfully before returning a REDELIVERY_EXHAUSTED error.

Use Secure Hash

Boolean

If true, Mule uses a secure hash algorithm to identify a redelivered message.

Message Digest Algorithm

String

Secure hashing algorithm to use if the Use Secure Hash field is true. If the payload of the message is a Java object, Mule ignores this value and returns the value that the payload’s hashCode() returned.

Id Expression

String

Defines one or more expressions to use to determine when a message has been redelivered. This property may only be set if Use Secure Hash is false.

Object Store

ObjectStore

Configures the object store that stores the redelivery counter for each message.

### Repeatable File Store Stream Configures the repeatable file-store streaming strategy by which Mule keeps a portion of the stream content in memory. If the stream content is larger than the configured buffer size, Mule backs up the buffer’s content to disk and then clears the memory.
FieldTypeDescriptionDefault ValueRequired

In Memory Size

Number

Maximum amount of memory that the stream can use for data. If the amount of memory exceeds this value, Mule buffers the content to disk. To optimize performance:

  • Configure a larger buffer size to avoid the number of times Mule needs to write the buffer on disk. This increases performance, but it also limits the number of concurrent requests your application can process, because it requires additional memory.

  • Configure a smaller buffer size to decrease memory load at the expense of response time.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the In Memory Size field.

### Repeatable In Memory Stream Configures the in-memory streaming strategy by which the request fails if the data exceeds the MAX buffer size. Always run performance tests to find the optimal buffer size for your specific use case.
FieldTypeDescriptionDefault ValueRequired

Initial Buffer Size

Number

Initial amount of memory to allocate to the data stream. If the streamed data exceeds this value, the buffer expands by Buffer Size Increment, with an upper limit of Max In Memory Size value.

Buffer Size Increment

Number

Amount by which the buffer size expands if it exceeds its initial size. Setting a value of 0 or lower specifies that the buffer can’t expand.

Max Buffer Size

Number

Maximum amount of memory to use. If more than that is used then a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value lower than or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the Initial Buffer Size, Buffer Size Increment, and Buffer Unit fields.

### Standard Revocation Check Configures standard revocation checks for TLS certificates.
FieldTypeDescriptionDefault ValueRequired

Only End Entities

Boolean

Which elements to verify in the certificate chain:

  • true

    Verify only the last element in the certificate chain.

  • false

    Verify all elements in the certificate chain.

Prefer Crls

Boolean

How to check certificate validity:

  • true

    Check the Certification Revocation List (CRL) for certificate validity.

  • false

    Use the Online Certificate Status Protocol (OCSP) to check certificate validity.

No Fallback

Boolean

Whether to use the secondary method to check certificate validity:

  • true

    Use the method that wasn’t specified in the Prefer Crls field (the secondary method) to check certificate validity.

  • false

    Do not use the secondary method to check certificate validity.

Soft Fail

Boolean

What to do if the revocation server can’t be reached or is busy:

  • true

    Avoid verification failure.

  • false

    Allow the verification to fail.

### Tcp Client Socket Params Configures the TCP client socket parameters.
FieldTypeDescriptionDefault ValueRequired

Send Buffer Size

Number

Size of the buffer (in bytes) used when sending data, set on the socket itself.

Receive Buffer Size

Number

Buffer size.

Client Timeout

Number

Sets the SO_TIMEOUT value on sockets. Indicates the amount of time (in milliseconds) that the socket waits in a blocking operation before failing. A value of 0 (the default) means waiting indefinitely.

0

Send Tcp No Delay

Boolean

If set, data is sent immediately rather than being collected for efficiency, prioritizing latency over network traffic reduction. Despite the socket default being false, this option defaults to true because reducing network traffic is generally not a major concern.

true

Linger

Number

Sets the SO_LINGER value in milliseconds. This determines how long the socket remains open to ensure all remaining data is transmitted correctly before closing.

Keep Alive

Boolean

Enables the SO_KEEPALIVE behavior on open sockets. This option periodically checks idle socket connections and closes them if they become unavailable. It is used by server sockets to control whether connections are kept alive before being recycled.

false

Connection Timeout

Number

How long the connector waits before timing out when establishing a connection to the remote service.

30000

### TLS Configures TLS to provide secure communications for the Mule app.
FieldTypeDescriptionDefault ValueRequired

Enabled Protocols

String

Comma-separated list of protocols enabled for this context.

Enabled Cipher Suites

String

Comma-separated list of cipher suites enabled for this context.

Trust Store

Configures the TLS truststore.

Key Store

Configures the TLS keystore.

Revocation Check

Configures a revocation checking mechanism.

### Topic Partition Type for topic partition.
FieldTypeDescriptionDefault ValueRequired

Topic

String

Topic name.

x

Partition

Number

Number of partitions.

x

### Truststore Configures the truststore for TLS.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the truststore. Mule resolves the path relative to the current classpath and file system.

Password

String

Password used to protect the truststore.

Type

String

Type of truststore.

Algorithm

String

Encryption algorithm that the truststore uses.

Insecure

Boolean

If true, Mule stops performing certificate validations. Setting this to true can make connections vulnerable to attacks.

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.14 --- # Apache Kafka Connector 4.14 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system and achieve seamless integration between your Mule app and a Kafka cluster, using Mule runtime engine (Mule). For compatibility information, see the Apache Kafka Connector Release Notes. ## Before You Begin Before creating an app, you must: - Have access to the Apache Kafka target resource and Anypoint Platform - Understand how to create a Mule app using Anypoint Studio - Have access to Apache Kafka to get values for the fields that appear in Studio ## Common Use Cases for the Connector Common use cases for Apache Kafka Connector include: - **Log Aggregation** Leverage the low-latency processing from Apache Kafka to collect logs from multiple services and make them available in a standard format to multiple consumers. For example, you can publish a message to Apache Kafka and then retrieve it. - **Analysis and Metrics** Optimize your advertising budget by integrating Apache Kafka and your big data analytics solution to analyze end user activity, such as page views, clicks, shares, and so on, to serve relevant ads. For example, a department store can use a website activity tracker to improve the online shopping experience. The gathered data is sent to multiple departments for various computations. Each department reviews the received information to stay informed about what the customer is looking for and then provides recommendations accordingly. - **Notifications and Alerts** Notify customers about various financial events, such as the recent amount of a transaction or more complex events like future investment suggestions based on integrations with credit agencies, location, and so on. - **Time-Sensitive Applications** Build integration applications that are time-sensitive. For example, you can build an application that ensures that patients in a hospital receive the care they need in a timely manner with a hospital server that processes emergency requests for patient admissions and orders the requests based on priority determined by specified criteria. In this scenario, an application processes Apache Kafka messages in the order that they are received, relying on the order and idempotency of the messages sent through the queue. For example, a newsroom can use the Apache Kafka system to deliver the latest news. Retrieve the latest news by building an integration application by reading from the end of the Apache Kafka queue first. For examples of these use cases, refer to [Apache Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-examples). ### Publish-to Use Cases With Kafka Connector, applications can publish binary messages to the Kafka server. Some examples of use cases include: - Publishing binary data to the Kafka server and the underlying topic. - Setting the batch size of the data to publish to Kafka when working with large volumes of data. - To optimize for performance, companies can batch messages to send. Kafka stores the message until the batch size is reached and completes the send at that time. - Configuring the message size to send to focus on optimal performance. - Kafka is not intended to process and handle large messages. Messages are typically in the 1 MB range and users set the size to avoid performance degradation. In the case of an error, the message is rejected. - Blocking further commits on a configurable level of acknowledgment for the messages published to the Kafka server to ensure successful data transmission. - Acknowledgments enable users to ensure that the message is sent prior to publishing more messages. This is specifically useful in time-ordered or event-ordered messages, such as logs or patient drug administration orders. Ensuring that messages are sent in order helps the downstream application process the messages in the right order. - Configuring how data is serialized during a publish. - Securing messages sent during Mutual TLS. The following lists the acceptance criteria for publishing: - Publish a message to a Kafka topic. - Configure the batch size and size of the message. - Obtain an acknowledgment once publishing is complete at the configured level. - Send the message securely via Mutual TLS. - Configure the serialization type. ### Subscribe-from Use Cases With Kafka Connector, applications can obtain messages from Kafka for further consumption downstream. Some examples of use cases include: Reading from Kafka: - Subscribing to the Kafka server and the underlying group to read and consume messages for downstream processing. - Subscribing to a specific Kafka partition to process messages. - Users with a high availability microservice don’t want to rely on Kafka’s rebalancing in the event of a failure. - If users have a local state associated with their partition, they want to process only that partition. - Configuring the offset value to a set value or to the beginning or end of the topic. - Users have a local store and want to use that information to start processing their records. - Users have time-sensitive records and want to skip to the end if the application must be up-to-date or skip to the beginning if it is a new application processing the records. - Reading the offset value from an alternate storage system on restart. - Users have a local store with the offset information and want to read the value from their local storage. - Setting the interval for which to block the receipt of messages for varying downstream processing intervals. Processing the messages: - Ensuring there is no data loss when the downstream application crashes by manually configuring commits to have more control. - Storing a commit value in a local store or in the Kafka storage and blocking until the store is successful. - Storing an offset asynchronously in a local store or in the Kafka storage. - Using Kafka’s in-built capability of load-balancing with multiple applications reading from the same group ID. - Configuring the number of records to process in a single call. The following lists the acceptance criteria for subscribing: - Read a message from Kafka from a specific partition or topic. - Read a message from Kafka after obtaining the offset from a local store. - Read a message from the beginning of the queue, the end of the queue, and a pre-specified offset. - Ensure a high level of performance when there is one consumer per thread. Multiple consumers exist in a single group. ## Audience - Starting user To create a Mule app, see [Apache Kafka Studio Configuration](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-studio). - Power user Read [XML and Maven Support](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-xml-maven) and [Examples](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-examples). ## Next After you complete the prerequisites, you are ready to create an app with [Anypoint Studio](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-studio). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Use Exchange to Discover Connectors, Templates, and Examples](https://docs.mulesoft.com/connectors/introduction/intro-use-exchange) - [Apache Kafka Connector](https://www.mulesoft.com/exchange/com.mulesoft.connectors/mule-kafka-connector/) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.14 --- # Apache Kafka Connector 4.14 - Additional Configuration Information > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Set additional configuration for Anypoint Connector for Apache Kafka (Apache Kafka Connector). ## Configure a Custom Partitioner You can configure a custom partitioner for Apache Kafka Connector in a Mule app: 1. Use Maven to create a Java project and add the custom partitioner to it: ```java public class CustomPartitioner implements Partitioner { // Implementation } ``` 2. Add this dependency in the Mule app: ```java org.example custom-partitioner 1.0-SNAPSHOT ``` 3. Add this dependency to `additionalPluginDependencies`: ```java org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 org.example custom-partitioner 1.0-SNAPSHOT mule-application ``` 4. Add the custom partitioner to the producer configuration: ```java ``` ## Next Step After you complete configuring the connector, you can try the [examples](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.14 --- # Apache Kafka Connector 4.14 Reference > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system. It provides seamless integration between your Mule app and an Apache Kafka cluster, using Mule runtime engine (Mule). > [!NOTE] When a **Message Listener** or a **Batch Message Listener** uses the `MANUAL` > acknowledgement mode, you must use a **Commit** operation at the end of the flow when the flow > finishes successfully (either because there were no errors or because there is an **On Error > Continue** component in the flow). > [!NOTE] Mule server notifications are not supported because the Kafka library used in the > connector manages network disruptions internally. ## Consumer Configuration Configuration for consumers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default acknowledgement mode

Enumeration, one of

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

AUTO

Default listener poll timeout

Number

Time to wait to perform a poll if data is not available in the buffer (fetched). If no value is set, the poll is returned immediately with any records that are currently available in the buffer or else returns empty if there is no data. Must not be negative.

100

Default listener poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default listener poll timeout field. This combines with Poll Timeout to define the total timeout for the polling.

MILLISECONDS

Default operation poll timeout

Number

Time to wait for an operation to finish. If no value is set or a negative value is set, the operation waits forever.

-1

Default operation poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default operation timeout field. This combines with Operation Timeout to define the total default timeout for the operations that use this configuration.

SECONDS

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Consumer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. Additionally, the connector supports only [RFC6749 standard](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4.3). For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Group ID

String

Default Group ID for all the Kafka Consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Whether internal topics matching a subscribed pattern is excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the reconnect backoff timeout scalar.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

true

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this field at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. You can override this field at the source level.

KB

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this field at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Request Timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This parameter can be overridden at source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Default record limit

Number

The maximum number of records returned on a poll call to the Kafka cluster. This parameter can be overridden at source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

The expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch heartbeat interval time scalar.

SECONDS

Session timeout

Number

The timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for session timeout scalar.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this config.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for connections maximum idle time scalar.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Topic Subscription Patterns

Array of String

The list of subscription regular expressions to subscribe to. This topics will be automatically rebalanced between the amount of consumers of the topic.

Assignments

The list of topic-partition pairs to assign. Note that there will be no automatic rebalance of the consumers

Default fetch minimum size

Number

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request waits for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency. This parameter can be overridden at source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

BYTE

Default fetch maximum size

Number

The maximum amount of data the server should return for a fetch request. Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). Note that the consumer performs multiple fetches in parallel. This parameter can be overridden at source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Default maximum partition fetch size

Number

The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size.This parameter can be overridden at source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Fetch Maximum Wait Timeout

Number

The maximum amount of time the server will block before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch maximum wait timeout scalar.

MILLISECONDS

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Consumer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Encryption type

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation. Enable FIPS mode by setting the `com.mulesoft.connectors.allow-registration-of-sasl-security-provider-in-fips=true` flag to true. This adds the `com.sun.security.sasl.Provider` security provider in the FIPS environment.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

User used by the client to connect to the Kafka broker.

x

Password

String

Password used by the client to connect to the Kafka broker.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

--- ## Producer Configuration Configuration for producers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default topic

String

Default topic name to use by the producer operations, overridable at the operation’s configuration level.

defaultTopicName

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Producer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names..

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Batch size

Number

The producer attempts to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size. Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent. A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the batch size scalar.

KB

Buffer size

Number

The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block for max.block.ms after which it will throw an exception. This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests. The default value in the Kafka docs is of 33554432 (32MB), but this should be capped to align with expected values for mule instances in cloudhub (v0.1 core)

1000

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

KB

DNS lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).

NONE

Connections maximum idle time

Number

Close idle connections after the value specified by this config.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the connections maximum idle scalar.

SECONDS

Delivery timeout

Number

An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retriable send failures. The producer may report failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch which reached an earlier delivery expiration deadline. The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the delivery timeout scalar.

SECONDS

Enable idempotence

Boolean

When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc, may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0 and acks must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConnectionException will be thrown

false

Linger time

Number

The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay?that is, rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it is sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the linger time scalar.

SECONDS

Maximum block time

Number

The configuration controls how long KafkaProducer.send() and KafkaProducer.partitionsFor() will block.These methods can be blocked either because the buffer is full or metadata unavailable.Blocking in the user-supplied serializers or partitioner will not be counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the maximum block time scalar.

SECONDS

Maximum in flight requests

Number

The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).

5

Maximum request size

Number

The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size which may be different from this.

1

Maximum request size unit.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

MB

Producer acknowledge mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent

NONE

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. This parameter can be overridden at source level.

KB

Retries amount

Number

Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first. Note additionally that produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires first before successful acknowledgment. Users should generally prefer to leave this config unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the retry backoff timeout time scalar.

MILLISECONDS

Retry backoff timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the request timeout time scalar.

SECONDS

Default request timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This should be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

The configuration controls which partitioning strategy is used when sending messages without providing a key or a partition.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Producer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

EncryptionType

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

## Consumer Sources - [Batch Message Listener](#batch-message-listener) - [Message Listener](#message-listener) ### Batch Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a list of messages to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Record

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) > [!NOTE] The **Batch Message Listener** source does not support configurable streaming strategies > because the source takes a batch of records as an input stream. The streaming strategy > configuration is non-repeatable-stream by default for the source. ### Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a single message to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) ## Consumer Operations - [Commit](#commit) - [Consume](#consume) - [Seek](#seek) ### Commit `` Commits the offsets associated to a message or batch of messages consumed in a **Message Listener**. This is a list or a single message consumed in the **Batch Message Listener** source.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumer commit key

String

Commit key of the last poll. This operation is valid only when used inside a flow that uses one of the sources (Batch Message Listener or Message Listener) which inserts this value as an attribute in the Mule event.

x

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_ACK\_MODE - KAFKA:RETRY\_EXHAUSTED - KAFKA:ALREADY\_COMMITTED - KAFKA:TIMEOUT - KAFKA:SESSION\_NOT\_FOUND - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Consume `` Receives messages from one or more Kafka topics. This operation works similarly to the **Message Listener** source, so all the operations that apply to that, apply to this operation as well. > [!NOTE] The **Consume** operation does not return the `consumerCommitKey`.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumption timeout

Number

Number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Consumption timeout field.

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

IMMEDIATE

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:SECURITY - KAFKA:RETRY\_EXHAUSTED - KAFKA:ILLEGAL\_STATE - KAFKA:TIMEOUT - KAFKA:SERIALIZATION\_ERROR - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Seek `` Sets the current offset of the consumer for the given topic and partition to the provided offset value.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which the Seek operation is performed.

x

Partition

Number

Partition number that has its offset modified.

x

Offset

Number

Offset value to commit for the configured partition.

x

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:INVALID\_TOPIC - KAFKA:SECURITY - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ## Producer Operations - [Bulk Publish](#bulkPublish) - [Publish](#publish) ### Bulk Publish `` Publishes a bulk of messages to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Messages

Array of Kafka Message

List of messages to publish.

#[payload]

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:AUTHENTICATION\_ERROR - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ### Publish `` Publishes a message to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Topic to publish to.

Partition

Number

(Optional) Topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ## Object Types - [Consumer Context](#ConsumerContext) - [CRL File](#CrlFile) - [Custom OCSP Responder](#CustomOcspResponder) - [Expiration Policy](#ExpirationPolicy) - [Kafka Message](#KafkaMessage) - [Kafka Message Attributes](#KafkaMessageAttributes) - [Kafka Message Metadata](#KafkaMessageMetadata) - [Kafka Record Attributes](#KafkaRecordAttributes) - [Keystore](#KeyStore) - [Proxy Config Params](#ProxyConfigParams) - [Reconnect](#Reconnect) - [Reconnect Forever](#Reconnect-Forever) - [Reconnection](#Reconnection) - [Record](#Record) - [Ser Byte Array](#SerByteArray) - [Redelivery Policy](#RedeliveryPolicy) - [Repeatable File Store Stream](#RepeatableFileStoreStream) - [Repeatable In Memory Stream](#RepeatableInMemoryStream) - [Standard Revocation Check](#StandardRevocationCheck) - [Tcp Client Socket Params](#TcpClientSocketParams) - [TLS](#Tls) - [Topic Partition](#TopicPartition) - [Truststore](#TrustStore) ### Consumer Context Type for the consumer context.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

### CRL File Specifies the location of the certification revocation list (CRL) file.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the CRL file.

### Custom OCSP Responder Configures a custom OCSP responder for certification revocation checks.
FieldTypeDescriptionDefault ValueRequired

Url

String

URL of the OCSP responder.

Cert Alias

String

Alias of the signing certificate for the OCSP response. If specified, the alias must be in the truststore.

### Expiration Policy Configures an expiration policy strategy.
FieldTypeDescriptionDefault ValueRequired

Max Idle Time

Number

Configures the maximum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Max Idle Time field.

### Kafka Message Represents a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Attributes

Attributes associated with the Kafka message.

Payload

Binary

Payload of the Kafka message, represented as an InputStream.

### Kafka Message Attributes Represents the attributes of a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Headers

Object

Headers of the Kafka message, represented as a map in which the key is a string and the value is an InputStream.

Key

Binary

Key of the Kafka message, represented as an InputStream.

Partition

Number

Partition of the topic where the Kafka message is sent.

Topic

String

Topic to which the Kafka message belongs to.

### Kafka Message Metadata Metadata of the Kafka message.
FieldTypeDescriptionDefault ValueRequired

Offset

Number

Offset of the message.

Partition

Number

Partition of the message.

Serialized Key Size

Number

Serialized key size of the message.

Serialized Value Size

Number

Serialized value size of the message.

Timestamp

DateTime

Timestamp of the message.

Topic

String

Topic of the message.

### Kafka Record Attributes Attributes of the Kafka record.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

Creation Timestamp

DateTime

Timestamp of record creation.

Headers

Object

Map of HTTP headers in the message.

Key

Binary

Keys of the record.

Leader Epoch

Number

Leader epoch of the record.

Log Append Timestamp

DateTime

Log append timestamp of the record.

Offset

Number

Offset of the record.

Partition

Number

Partition of the record.

Serialized Key Size

Number

Serialized key size of the record.

Serialized Value Size

Number

Serialized value size of the record.

Topic

String

Topic of the record.

### Keystore Configures the keystore for the TLS protocol. The keystore you generate contains a private key and a public certificate.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the keystore. Mule resolves the path relative to the current classpath and file system.

Type

String

Type of keystore.

Alias

String

Alias of the key to use when the keystore contains multiple private keys. By default, Mule uses the first key in the file.

Key Password

String

Password used to protect the private key.

Password

String

Password used to protect the keystore.

Algorithm

String

Encryption algorithm that the keystore uses.

### Proxy Config Params Configures the proxy configuration parameters.
FieldTypeDescriptionDefault ValueRequired

Host

String

Hostname or IP address of the proxy server.

Port

Number

Port of the proxy server.

Username

String

Username to authenticate against the proxy server.

Password

String

Password to authenticate against the proxy server.

### Reconnect Configures a standard reconnection strategy, which specifies how often to reconnect and how many reconnection attempts the connector source or operation can make.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

Count

Number

How many reconnection attempts the Mule app can make.

### Reconnect Forever Configures a forever reconnection strategy by which the connector source or operation attempts to reconnect at a specified frequency for as long as the Mule app runs.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

### Reconnection Configures a reconnection strategy for an operation.
FieldTypeDescriptionDefault ValueRequired

Fails Deployment

Boolean

What to do if, when an app is deployed, a connectivity test does not pass after exhausting the associated reconnection strategy:

  • true

    Allow the deployment to fail.

  • false

    Ignore the results of the connectivity test.

Reconnection Strategy

Reconnection strategy to use.

### Record Type for records.
FieldTypeDescriptionDefault ValueRequired

Attributes

Kafka record attributes.

Payload

Binary

Payload for the record.

Ser Byte Array

### Ser Byte Array
FieldTypeDescriptionDefault ValueRequired

Input Stream

Binary

### Redelivery Policy Configures the redelivery policy for executing requests that generate errors. You can add a redelivery policy to any source in a flow.
FieldTypeDescriptionDefault ValueRequired

Max Redelivery Count

Number

Maximum number of times that a redelivered request can be processed unsuccessfully before returning a REDELIVERY_EXHAUSTED error.

Use Secure Hash

Boolean

If true, Mule uses a secure hash algorithm to identify a redelivered message.

Message Digest Algorithm

String

Secure hashing algorithm to use if the Use Secure Hash field is true. If the payload of the message is a Java object, Mule ignores this value and returns the value that the payload’s hashCode() returned.

Id Expression

String

Defines one or more expressions to use to determine when a message has been redelivered. This property may only be set if Use Secure Hash is false.

Object Store

ObjectStore

Configures the object store that stores the redelivery counter for each message.

### Repeatable File Store Stream Configures the repeatable file-store streaming strategy by which Mule keeps a portion of the stream content in memory. If the stream content is larger than the configured buffer size, Mule backs up the buffer’s content to disk and then clears the memory.
FieldTypeDescriptionDefault ValueRequired

In Memory Size

Number

Maximum amount of memory that the stream can use for data. If the amount of memory exceeds this value, Mule buffers the content to disk. To optimize performance:

  • Configure a larger buffer size to avoid the number of times Mule needs to write the buffer on disk. This increases performance, but it also limits the number of concurrent requests your application can process, because it requires additional memory.

  • Configure a smaller buffer size to decrease memory load at the expense of response time.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the In Memory Size field.

### Repeatable In Memory Stream Configures the in-memory streaming strategy by which the request fails if the data exceeds the MAX buffer size. Always run performance tests to find the optimal buffer size for your specific use case.
FieldTypeDescriptionDefault ValueRequired

Initial Buffer Size

Number

Initial amount of memory to allocate to the data stream. If the streamed data exceeds this value, the buffer expands by Buffer Size Increment, with an upper limit of Max In Memory Size value.

Buffer Size Increment

Number

Amount by which the buffer size expands if it exceeds its initial size. Setting a value of 0 or lower specifies that the buffer can’t expand.

Max Buffer Size

Number

Maximum amount of memory to use. If more than that is used then a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value lower than or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the Initial Buffer Size, Buffer Size Increment, and Buffer Unit fields.

### Standard Revocation Check Configures standard revocation checks for TLS certificates.
FieldTypeDescriptionDefault ValueRequired

Only End Entities

Boolean

Which elements to verify in the certificate chain:

  • true

    Verify only the last element in the certificate chain.

  • false

    Verify all elements in the certificate chain.

Prefer Crls

Boolean

How to check certificate validity:

  • true

    Check the Certification Revocation List (CRL) for certificate validity.

  • false

    Use the Online Certificate Status Protocol (OCSP) to check certificate validity.

No Fallback

Boolean

Whether to use the secondary method to check certificate validity:

  • true

    Use the method that wasn’t specified in the Prefer Crls field (the secondary method) to check certificate validity.

  • false

    Do not use the secondary method to check certificate validity.

Soft Fail

Boolean

What to do if the revocation server can’t be reached or is busy:

  • true

    Avoid verification failure.

  • false

    Allow the verification to fail.

### Tcp Client Socket Params Configures the TCP client socket parameters.
FieldTypeDescriptionDefault ValueRequired

Send Buffer Size

Number

Size of the buffer (in bytes) used when sending data, set on the socket itself.

Receive Buffer Size

Number

Buffer size.

Client Timeout

Number

Sets the SO_TIMEOUT value on sockets. Indicates the amount of time (in milliseconds) that the socket waits in a blocking operation before failing. A value of 0 (the default) means waiting indefinitely.

0

Send Tcp No Delay

Boolean

If set, data is sent immediately rather than being collected for efficiency, prioritizing latency over network traffic reduction. Despite the socket default being false, this option defaults to true because reducing network traffic is generally not a major concern.

true

Linger

Number

Sets the SO_LINGER value in milliseconds. This determines how long the socket remains open to ensure all remaining data is transmitted correctly before closing.

Keep Alive

Boolean

Enables the SO_KEEPALIVE behavior on open sockets. This option periodically checks idle socket connections and closes them if they become unavailable. It is used by server sockets to control whether connections are kept alive before being recycled.

false

Connection Timeout

Number

How long the connector waits before timing out when establishing a connection to the remote service.

30000

### TLS Configures TLS to provide secure communications for the Mule app.
FieldTypeDescriptionDefault ValueRequired

Enabled Protocols

String

Comma-separated list of protocols enabled for this context.

Enabled Cipher Suites

String

Comma-separated list of cipher suites enabled for this context.

Trust Store

Configures the TLS truststore.

Key Store

Configures the TLS keystore.

Revocation Check

Configures a revocation checking mechanism.

### Topic Partition Type for topic partition.
FieldTypeDescriptionDefault ValueRequired

Topic

String

Topic name.

x

Partition

Number

Number of partitions.

x

### Truststore Configures the truststore for TLS.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the truststore. Mule resolves the path relative to the current classpath and file system.

Password

String

Password used to protect the truststore.

Type

String

Type of truststore.

Algorithm

String

Encryption algorithm that the truststore uses.

Insecure

Boolean

If true, Mule stops performing certificate validations. Setting this to true can make connections vulnerable to attacks.

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Apache Kafka Connector 4.15 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system and achieve seamless integration between your Mule app and a Kafka cluster, using Mule runtime engine (Mule). For compatibility information, see the Apache Kafka Connector Release Notes. ## Before You Begin Before creating an app, you must: - Have access to the Apache Kafka target resource and Anypoint Platform - Understand how to create a Mule app using Anypoint Studio - Have access to Apache Kafka to get values for the fields that appear in Studio ## Common Use Cases for the Connector Common use cases for Apache Kafka Connector include: - **Log Aggregation** Leverage the low-latency processing from Apache Kafka to collect logs from multiple services and make them available in a standard format to multiple consumers. For example, you can publish a message to Apache Kafka and then retrieve it. - **Analysis and Metrics** Optimize your advertising budget by integrating Apache Kafka and your big data analytics solution to analyze end user activity, such as page views, clicks, shares, and so on, to serve relevant ads. For example, a department store can use a website activity tracker to improve the online shopping experience. The gathered data is sent to multiple departments for various computations. Each department reviews the received information to stay informed about what the customer is looking for and then provides recommendations accordingly. - **Notifications and Alerts** Notify customers about various financial events, such as the recent amount of a transaction or more complex events like future investment suggestions based on integrations with credit agencies, location, and so on. - **Time-Sensitive Applications** Build integration applications that are time-sensitive. For example, you can build an application that ensures that patients in a hospital receive the care they need in a timely manner with a hospital server that processes emergency requests for patient admissions and orders the requests based on priority determined by specified criteria. In this scenario, an application processes Apache Kafka messages in the order that they are received, relying on the order and idempotency of the messages sent through the queue. For example, a newsroom can use the Apache Kafka system to deliver the latest news. Retrieve the latest news by building an integration application by reading from the end of the Apache Kafka queue first. For examples of these use cases, refer to [Apache Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-examples). ### Publish-to Use Cases With Kafka Connector, applications can publish binary messages to the Kafka server. Some examples of use cases include: - Publishing binary data to the Kafka server and the underlying topic. - Setting the batch size of the data to publish to Kafka when working with large volumes of data. - To optimize for performance, companies can batch messages to send. Kafka stores the message until the batch size is reached and completes the send at that time. - Configuring the message size to send to focus on optimal performance. - Kafka is not intended to process and handle large messages. Messages are typically in the 1 MB range and users set the size to avoid performance degradation. In the case of an error, the message is rejected. - Blocking further commits on a configurable level of acknowledgment for the messages published to the Kafka server to ensure successful data transmission. - Acknowledgments enable users to ensure that the message is sent prior to publishing more messages. This is specifically useful in time-ordered or event-ordered messages, such as logs or patient drug administration orders. Ensuring that messages are sent in order helps the downstream application process the messages in the right order. - Configuring how data is serialized during a publish. - Securing messages sent during Mutual TLS. The following lists the acceptance criteria for publishing: - Publish a message to a Kafka topic. - Configure the batch size and size of the message. - Obtain an acknowledgment once publishing is complete at the configured level. - Send the message securely via Mutual TLS. - Configure the serialization type. ### Subscribe-from Use Cases With Kafka Connector, applications can obtain messages from Kafka for further consumption downstream. Some examples of use cases include: Reading from Kafka: - Subscribing to the Kafka server and the underlying group to read and consume messages for downstream processing. - Subscribing to a specific Kafka partition to process messages. - Users with a high availability microservice don’t want to rely on Kafka’s rebalancing in the event of a failure. - If users have a local state associated with their partition, they want to process only that partition. - Configuring the offset value to a set value or to the beginning or end of the topic. - Users have a local store and want to use that information to start processing their records. - Users have time-sensitive records and want to skip to the end if the application must be up-to-date or skip to the beginning if it is a new application processing the records. - Reading the offset value from an alternate storage system on restart. - Users have a local store with the offset information and want to read the value from their local storage. - Setting the interval for which to block the receipt of messages for varying downstream processing intervals. Processing the messages: - Ensuring there is no data loss when the downstream application crashes by manually configuring commits to have more control. - Storing a commit value in a local store or in the Kafka storage and blocking until the store is successful. - Storing an offset asynchronously in a local store or in the Kafka storage. - Using Kafka’s in-built capability of load-balancing with multiple applications reading from the same group ID. - Configuring the number of records to process in a single call. The following lists the acceptance criteria for subscribing: - Read a message from Kafka from a specific partition or topic. - Read a message from Kafka after obtaining the offset from a local store. - Read a message from the beginning of the queue, the end of the queue, and a pre-specified offset. - Ensure a high level of performance when there is one consumer per thread. Multiple consumers exist in a single group. ## Audience - Starting user To create a Mule app, see [Apache Kafka Studio Configuration](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-studio). - Power user Read [XML and Maven Support](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-xml-maven) and [Examples](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-examples). ## Next After you complete the prerequisites, you are ready to create an app with [Anypoint Studio](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-studio). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Use Exchange to Discover Connectors, Templates, and Examples](https://docs.mulesoft.com/connectors/introduction/intro-use-exchange) - [Apache Kafka Connector](https://www.mulesoft.com/exchange/com.mulesoft.connectors/mule-kafka-connector/) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Apache Kafka Connector 4.15 - Additional Configuration Information > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Set additional configuration for Anypoint Connector for Apache Kafka (Apache Kafka Connector). ## Configure a Custom Partitioner You can configure a custom partitioner for Apache Kafka Connector in a Mule app: 1. Use Maven to create a Java project and add the custom partitioner to it: ```java public class CustomPartitioner implements Partitioner { // Implementation } ``` 2. Add this dependency in the Mule app: ```java org.example custom-partitioner 1.0-SNAPSHOT ``` 3. Add this dependency to `additionalPluginDependencies`: ```java org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 org.example custom-partitioner 1.0-SNAPSHOT mule-application ``` 4. Add the custom partitioner to the producer configuration: ```java ``` ## Next Step After you complete configuring the connector, you can try the [examples](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-examples). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Apache Kafka Connector 4.15 Reference > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Apache Kafka (Apache Kafka Connector) enables you to interact with the Apache Kafka messaging system. It provides seamless integration between your Mule app and an Apache Kafka cluster, using Mule runtime engine (Mule). > [!NOTE] When a **Message Listener** or a **Batch Message Listener** uses the `MANUAL` > acknowledgement mode, you must use a **Commit** operation at the end of the flow when the flow > finishes successfully (either because there were no errors or because there is an **On Error > Continue** component in the flow). > [!NOTE] Mule server notifications are not supported because the Kafka library used in the > connector manages network disruptions internally. ## Consumer Configuration Configuration for consumers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default acknowledgement mode

Enumeration, one of

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

AUTO

Default listener poll timeout

Number

Time to wait to perform a poll if data is not available in the buffer (fetched). If no value is set, the poll is returned immediately with any records that are currently available in the buffer or else returns empty if there is no data. Must not be negative.

100

Default listener poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default listener poll timeout field. This combines with Poll Timeout to define the total timeout for the polling.

MILLISECONDS

Default operation poll timeout

Number

Time to wait for an operation to finish. If no value is set or a negative value is set, the operation waits forever.

-1

Default operation poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default operation timeout field. This combines with Operation Timeout to define the total default timeout for the operations that use this configuration.

SECONDS

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Consumer SASL/AWS IAM Connection Type Use AWS Identity and Access Management (IAM) authentication to connect to an Amazon MSK cluster.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster.

x

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled.

Group ID

String

Default group ID for all Kafka consumers that use this configuration.

Consumer Amount

Number

Number of consumers that the connection initially creates.

1

Maximum polling interval

Number

Maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the time that a consumer can be idle before fetching more records. If poll() isn’t called before this timeout expires, the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id that reach this timeout, partitions aren’t immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after session.timeout.ms expires.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, including transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are returned in offset order. In READ_COMMITTED mode, consumer.poll() returns messages only up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

Messages after those belonging to ongoing transactions are withheld until the relevant transaction completes. As a result, READ_COMMITTED consumers can’t read to the high watermark when transactions are in flight. In READ_COMMITTED mode, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response isn’t received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS, when the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. This applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY, each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats ensure that the consumer’s session stays active and facilitate rebalancing when consumers join or leave the group. The value must be lower than session.timeout.ms, but typically should be no higher than 1/3 of that value. You can adjust it lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate that it is active to the broker. If no heartbeats are received before the session timeout expires, the broker removes the consumer from the group and initiates a rebalance. The value must be in the allowable range configured by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the specified number of milliseconds.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

TLS Configuration

Defines a TLS configuration that both clients and servers can use to secure communication for the Mule app. The connector automatically sets the security.protocol used for communication. Valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL. The default is PLAINTEXT, or SASL_PLAINTEXT for SASL authentication. If the broker uses SSL, configure at least the keystore in the tls:context child element. The connector then automatically uses SSL, or SASL_SSL for SASL authentication, as the security.protocol.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers aren’t automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this isn’t an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Access Key ID

String

AWS access key ID of the principal that can assume the IAM role. Leave this field empty to use the default credentials provider chain.

Secret Access Key

String

AWS secret access key of the principal that can assume the IAM role. Leave this field empty to use the default credentials provider chain.

Session Token

String

AWS session token. Specify this field only when the caller credentials are temporary, such as federated credentials, AWS SSO credentials, or credentials from a previous STS call.

Role ARN

String

ARN of the IAM role to assume for MSK access through STS AssumeRole. This field is optional when using the default credentials provider chain without assuming a role.

Role Session Name

String

Name attached to the assumed-role session. This name appears in AWS CloudTrail audit logs.

mule-kafka-connector

AWS Region

String

AWS Region where the MSK cluster is located. This field also determines STS endpoint routing.

x

Try Default AWS Credentials Provider Chain

Boolean

Use the AWS default credentials provider chain instead of an explicit access key and secret access key. This works only when an AWS identity is attached to the runtime.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. Additionally, the connector supports only [RFC6749 standard](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4.3). For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Group ID

String

Default Group ID for all the Kafka Consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Whether internal topics matching a subscribed pattern is excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the reconnect backoff timeout scalar.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

true

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this field at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. You can override this field at the source level.

KB

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this field at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Request Timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This parameter can be overridden at source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for request timeout scalar. This parameter can be overridden at source level.

SECONDS

Default record limit

Number

The maximum number of records returned on a poll call to the Kafka cluster. This parameter can be overridden at source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

The expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch heartbeat interval time scalar.

SECONDS

Session timeout

Number

The timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for session timeout scalar.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this config.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for connections maximum idle time scalar.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Topic Subscription Patterns

Array of String

The list of subscription regular expressions to subscribe to. This topics will be automatically rebalanced between the amount of consumers of the topic.

Assignments

The list of topic-partition pairs to assign. Note that there will be no automatic rebalance of the consumers

Default fetch minimum size

Number

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request waits for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency. This parameter can be overridden at source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

BYTE

Default fetch maximum size

Number

The maximum amount of data the server should return for a fetch request. Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). Note that the consumer performs multiple fetches in parallel. This parameter can be overridden at source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Default maximum partition fetch size

Number

The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size.This parameter can be overridden at source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the maximum partition fetch size scalar. This parameter can be overridden at source level.

MB

Fetch Maximum Wait Timeout

Number

The maximum amount of time the server will block before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for fetch maximum wait timeout scalar.

MILLISECONDS

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Consumer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Encryption type

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation. Enable FIPS mode by setting the `com.mulesoft.connectors.allow-registration-of-sasl-security-provider-in-fips=true` flag to true. This adds the `com.sun.security.sasl.Provider` security provider in the FIPS environment.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Username

String

User used by the client to connect to the Kafka broker.

x

Password

String

Password used by the client to connect to the Kafka broker.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Consumer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Group ID

String

Default group ID for all the Kafka consumers that use this configuration.

Consumer Amount

Number

Determines the number of consumers the connection initially creates.

1

Maximum polling interval

Number

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before the expiration of this timeout, then the consumer is considered failed and the group rebalances to reassign the partitions to another member. For consumers using a non-null group.instance.id which reaches this timeout, partitions are not immediately reassigned. Instead, the consumer stops sending heartbeats and partitions are reassigned after the expiration of session.timeout.ms.

60

Maximum Polling Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum polling interval field. You can override this parameter at the source level.

SECONDS

Isolation Level

Enumeration, one of:

  • READ_UNCOMMITTED

  • READ_COMMITTED

Controls how to read messages that are written transactionally.

  • READ_COMMITTED

    consumer.poll() returns only committed transactional messages are returned.

  • READ_UNCOMMITTED

    consumer.poll() returns all messages, even transactional messages that were aborted.

Non-transactional messages are returned unconditionally in either mode. Messages are always returned in offset order. Hence, in the READ_COMMITTED mode, consumer.poll() returns only messages up to the last stable offset (LSO), which is one less than the offset of the first open transaction.

In particular, any messages appearing after messages belonging to ongoing transactions are withheld until the relevant transaction is completed. As a result, READ_COMMITTED consumers are not able to read up to the high watermark when there are in-flight transactions. Furthermore, when in READ_COMMITTED, the seekToEnd method returns the LSO.

READ_UNCOMMITTED

Exclude internal topics

Boolean

Determines whether internal topics matching a subscribed pattern are excluded from the subscription. It is always possible to explicitly subscribe to an internal topic.

true

Auto offset reset

Enumeration, one of:

  • EARLIEST

  • LATEST

  • ERROR

Determines what to do when there is no initial offset in Kafka or if the current offset no longer exists on the server, for example, because the data was deleted.

  • EARLIEST

    Automatically reset the offset to the earliest offset.

  • LATEST

    Automatically reset the offset to the latest offset.

  • ERROR

    Throw an error if no previous offset is found for the consumer’s group.

LATEST

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Check CRC

Boolean

Automatically check the CRC32 of the records consumed. This ensures that no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so in situations that require extremely high performance, this can be disabled.

true

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Request Timeout

Number

Maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses, the client resends the request if necessary, or fails the request if the retries are exhausted. You can override this parameter at the source level.

30

Request Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Request Timeout field. You can override this parameter at the source level.

SECONDS

Default record limit

Number

Maximum number of records returned on a poll call to the Kafka cluster. You can override this parameter at the source level.

500

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Heartbeat interval

Number

Expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the consumer’s session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

3

Heartbeat Interval Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Heartbeat interval field.

SECONDS

Session Timeout

Number

Timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its aliveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker removes this consumer from the group and initiates a rebalance. The value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

10

Session timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Session Timeout field.

SECONDS

Connection maximum idle time

Number

Close idle connections after the number of milliseconds specified by this configuration.

540

Connection maximum idle time time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connection maximum idle time field.

SECONDS

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. The default value is PLAINTEXT or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain. If the broker configures SSL as the protocol, then configure at least the keystore in the tls:context child element of the configuration and the connector automatically uses SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Topic Subscription Patterns

Array of String

List of subscription regular expressions to which to subscribe. Topics are automatically rebalanced between the topic consumers.

Assignments

List of topic-partition pairs to assign. Consumers are not automatically rebalanced.

Default fetch minimum size

Number

Minimum amount of data the server returns for a fetch request. If insufficient data is available, the request waits for the specified minimum amount of data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. If you set this to a value greater than 1, the server waits for larger amounts of data to accumulate, which can improve server throughput slightly at the cost of some additional latency. You can override this parameter at the source level.

1

Fetch Minimum Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch minimum size field. You can override this parameter at the source level.

BYTE

Default fetch maximum size

Number

Maximum amount of data the server returns for a fetch request. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch is still returned to ensure that the consumer can make progress. As such, this is not an absolute maximum. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). The consumer performs multiple fetches in parallel. You can override this parameter at the source level.

1

Default maximum fetch size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default fetch maximum size field. You can override this parameter at the source level.

MB

Default maximum partition fetch size

Number

Maximum amount of data per partition that the server returns. The consumer fetches records in batches. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch is still returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined using message.max.bytes (broker configuration) or max.message.bytes (topic configuration). See fetch.max.bytes for limiting the consumer request size. You can override this parameter at the source level.

1

Default maximum partition fetch unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default maximum partition fetch size field. You can override this parameter at the source level.

MB

Fetch Maximum Wait Timeout

Number

Maximum amount of time the server blocks before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement specified by fetch.min.bytes.

500

Fetch Maximum Wait Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Fetch Maximum Wait Timeout field.

MILLISECONDS

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

--- ## Producer Configuration Configuration for producers for Apache Kafka Connector.
NameTypeDescriptionDefault ValueRequired

Name

String

Name for this configuration. Connectors reference the configuration with this name.

x

Connection

Connection types for this configuration.

x

Default topic

String

Default topic name to use by the producer operations, overridable at the operation’s configuration level.

defaultTopicName

Zone ID

String

Converts the provided timestamps into ZonedLocalDateTimes in the results. The default value is provided by the system.

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

### Producer Plaintext Connection Type Use an unauthenticated and non-encrypted connection type.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names..

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/AWS IAM Connection Type Use AWS Identity and Access Management (IAM) authentication to connect to an Amazon MSK cluster.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster.

x

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the broker’s certificate.

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput. A batch size of 0 disables batching entirely. A very large batch size can result in more wasteful use of memory because a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but isn’t exact because not all producer memory is used for buffering. Some additional memory is used for compression, if enabled, and for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32 MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1000

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS, when the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. This applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY, each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed before sending, the time to await acknowledgment from the broker, and the time allowed for retrying send failures. The producer might report failure earlier than this configuration if it encounters an unrecoverable error, retries are exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. This configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of retried messages in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0, and acks to be all. If these values aren’t explicitly set, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together records that arrive between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent. This setting adds a small artificial delay, allowing more records to be sent in the same batch. It gives the upper bound on the delay for batching. Once the specified batch.size worth of records is received for a partition, the batch is sent immediately. If fewer than the specified number of bytes is received, the producer waits for the specified time for more records to arrive. This setting defaults to 0 (no delay). Setting linger.ms=5, for example, reduces the number of requests sent but can add up to 5 ms of latency to records sent without load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls how long KafkaProducer.send() and KafkaProducer.partitionsFor() can block. These methods can be blocked because the buffer is full or metadata is unavailable. Blocking in user-supplied serializers or partitioners isn’t counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and sends fail, there is a risk of message reordering due to retries.

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. It also caps the maximum record batch size. The server has its own cap on record batch size, which might be different from this value.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Allowing retries without setting max.in.flight.requests.per.connection to 1 can change the ordering of records. If two batches are sent to a single partition, the first fails and is retried but the second succeeds, the records in the second batch might appear first. Produce requests can fail before retries are exhausted if the timeout configured by delivery.timeout.ms expires. It is recommended that users leave this configuration unset and use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Retry backoff timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field.

KB

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response isn’t received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms to reduce the possibility of message duplication due to unnecessary producer retries.

30

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

TLS Configuration

Defines a TLS configuration that both clients and servers can use to secure communication for the Mule app. The connector automatically sets the security.protocol used for communication. Valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL. The default is PLAINTEXT, or SASL_PLAINTEXT for SASL authentication. If the broker uses SSL, configure at least the keystore in the tls:context child element. The connector then automatically uses SSL, or SASL_SSL for SASL authentication, as the security.protocol.

Access Key ID

String

AWS access key ID of the principal that can assume the IAM role. Leave this field empty to use the default credentials provider chain.

Secret Access Key

String

AWS secret access key of the principal that can assume the IAM role. Leave this field empty to use the default credentials provider chain.

Session Token

String

AWS session token. Specify this field only when the caller credentials are temporary, such as federated credentials, AWS SSO credentials, or credentials from a previous STS call.

Role ARN

String

ARN of the IAM role to assume for MSK access through STS AssumeRole. This field is optional when using the default credentials provider chain without assuming a role.

Role Session Name

String

Name attached to the assumed-role session. This name appears in AWS CloudTrail audit logs.

mule-kafka-connector

AWS Region

String

AWS Region where the MSK cluster is located. This field also determines STS endpoint routing.

x

Try Default AWS Credentials Provider Chain

Boolean

Use the AWS default credentials provider chain instead of an explicit access key and secret access key. This works only when an AWS identity is attached to the runtime.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer Kerberos Connection Type Use Kerberos configuration files.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Principal

String

Entity that is authenticated by a computer system or a network. Principals can be individual people, computers, services, or computational entities such as processes and threads.

x

Service name

String

Kerberos principal name that Kafka runs as.

x

Kerberos configuration file (krb5.conf)

String

Path to the krb5.conf file, which contains Kerberos configuration information. This information includes the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm, defaults for Kerberos applications, and the mappings of hostnames to Kerberos realms.

Use ticket cache

Boolean

Set this option to true to obtain the ticket-granting ticket (TGT) from the ticket cache. Set this option to false if you do not want to use the ticket cache. The connector searches for the ticket cache as follows:

  • On Solaris and Linux, the connector looks in /tmp/krb5cc_uid, in which the uid is the numeric user identifier.

  • If the ticket cache is not available in /tmp/krb5cc_uid or the app is on a Windows platform, the connector looks in {user.home}{file.separator}krb5cc_{user.name}. You can override the ticket cache location by setting a value for the Ticket cache field.

    In a Windows environment, if a ticket cannot be retrieved from the file ticket cache, Windows uses the Local Security Authority (LSA) API to get the ticket-granting ticket (TGT).

false

Ticket cache

String

Name of the ticket cache that contains the user’s ticket-granting ticket (TGT). If this value is set, Use ticket cache must also be set to true. Otherwise, a configuration error is returned.

Use keytab

Boolean

Set this option to true if you want the connector to obtain the principal’s key from the keytab. If you don’t set this value, the connector locates the keytab by using the Kerberos configuration file. If the keytab is not specified in the Kerberos configuration file, the connector looks for the {user.home}{file.separator}krb5.keytab file.

false

Keytab

String

Set this option to the file name of the keytab to obtain the principal’s secret key.

Store key

Boolean

Set this option to true to store the principal’s subject private credentials.

false

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/OAUTHBEARER - Client Credentials Connection Type OAuth Bearer authentication is a mechanism for authenticating requests using bearer tokens, which are typically used to authenticate users and applications to access resources. Apache Kafka® supports OAuth Bearer authentication for establishing connections securely. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Endpoint identification algorithm

String

The endpoint identification algorithm used by clients to validate server host name. The default value is an empty string, which means it is disabled. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the brokers certificate.

Batch size

Number

The producer attempts to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size. Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent. A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the batch size scalar.

KB

Buffer size

Number

The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block for max.block.ms after which it will throw an exception. This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests. The default value in the Kafka docs is of 33554432 (32MB), but this should be capped to align with expected values for mule instances in cloudhub (v0.1 core)

1000

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

KB

DNS lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to use_all_dns_ips then, when the lookup returns multiple IP addresses for a hostname, they will all be attempted to connect to before failing the connection. Applies to both bootstrap and advertised servers. If the value is resolve_canonical_bootstrap_servers_only each entry will be resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).

NONE

Connections maximum idle time

Number

Close idle connections after the value specified by this config.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the connections maximum idle scalar.

SECONDS

Delivery timeout

Number

An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retriable send failures. The producer may report failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch which reached an earlier delivery expiration deadline. The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the delivery timeout scalar.

SECONDS

Enable idempotence

Boolean

When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc, may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0 and acks must be 'all'. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConnectionException will be thrown

false

Linger time

Number

The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay?that is, rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch.size worth of records for a partition it is sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the linger time scalar.

SECONDS

Maximum block time

Number

The configuration controls how long KafkaProducer.send() and KafkaProducer.partitionsFor() will block.These methods can be blocked either because the buffer is full or metadata unavailable.Blocking in the user-supplied serializers or partitioner will not be counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the maximum block time scalar.

SECONDS

Maximum in flight requests

Number

The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).

5

Maximum request size

Number

The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size which may be different from this.

1

Maximum request size unit.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the max request size scalar.

MB

Producer acknowledge mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent

NONE

Default receive buffer size

Number

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the receive buffer size scalar. This parameter can be overridden at source level.

KB

Retries amount

Number

Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first. Note additionally that produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires first before successful acknowledgment. Users should generally prefer to leave this config unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the retry backoff timeout time scalar.

MILLISECONDS

Retry backoff timeout

Number

The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Default send buffer size

Number

The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used. This parameter can be overridden at source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit of measure for the send buffer size scalar. This parameter can be overridden at source level.

KB

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Determines the time unit for the request timeout time scalar.

SECONDS

Default request timeout

Number

The configuration controls the maximum amount of time the client waits for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted. This should be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

The configuration controls which partitioning strategy is used when sending messages without providing a key or a partition.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, which can be used from both the client and server sides to secure communication for the Mule app. The connector will automatically set the 'security.protocol' to use for communication. Valid values are PLAINTEXT / SSL / SASL_PLAINTEXT / SASL_SSL. Default value when no configuration has been provided is PLAINTEXT(or SASL_PLAINTEXT for SASL authentication - kerberos/scram/plain/token). If SSL was configured as protocol on the broker side then the user needs to configure at least the keystore in the 'tls:context' child element of this config and the connector will automatically use SSL(or SASL_SSL for SASL authentication) as 'security.protocol'.

Client ID Key

String

Specifies the configuration key used to define the OAuth client ID within the JAAS configuration.

clientId

Client ID

String

The value for the Client ID which refers to the unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes.

x

Client Secret Key

String

Specifies the configuration key used to define the OAuth client secret within the JAAS configuration.

clientSecret

Client Secret

String

The value for the client secret which refers to the confidential authentication credential assigned to the OAuth client.

x

OAuth Token Endpoint Key

String

Specifies the configuration key used to define the OAuth token endpoint within the JAAS configuration.

sasl.oauthbearer.token.endpoint.url

OAuth Token Endpoint

String

Describes the URL where clients can request access tokens from the authentication server.

x

Scope Key

String

Specifies the configuration key used to define the OAuth scope within the JAAS configuration.

scope

Scope

String

Defines the specific permissions and access rights granted to the client application by the authorization server.

Audience Key

String

Specifies the configuration key used to define the OAuth audience within the JAAS configuration.

audience

Audience

String

Comma separated list of one or more target audiences for the access token.

Max Token Expiration Seconds

Number

Sets the maximum duration, in seconds, for which an access token remains valid. This helps prevent unauthorized access by limiting the access token’s lifespan, ensuring that even if the access token is intercepted, it becomes invalid after a set period, thus enhancing security and enforcing re-authentication.

The default value is null (not set). When this field is not configured, the connector uses the actual token expiration time provided by the OAuth server in the token response (expires_in field). The token is valid for its full natural lifespan as determined by the authorization server. Leaving this field unset uses the OAuth server’s intended token lifespan, balances security and performance, and follows best practices.

If this value is set longer than the actual token TTL, the token expires at its natural expiration time as provided by the OAuth server, not at the configured maximum. If this value is set to the same value as the actual token TTL, the token expires at the configured time, which happens to match the server’s intended expiration time.

Principal Name

String

Unique identifier associated with an authenticated user, often used for authorization decisions and resource access.

Credentials Placement

Enumeration, one of:

  • BASIC_AUTHENTICATION_HEADER

  • BODY

Defines whether to include client credentials in the basic authentication header or in the body of the authentication request.

BASIC_AUTHENTICATION_HEADER

OAuth Module Required

Boolean

Indicates whether this login module must be successfully authenticated for Kafka clients to establish a connection using OAuth 2.0 bearer tokens.

true

Include Accept Header

Boolean

Whether to include 'Accept-application/json' header in the authentication request.

true

OAuth Extensions

Object

Represents a mapping of key-value pairs containing extensions for custom broker OAuth implementations. If you specify a key for the OAuth Extensions field, ensure that the input contains only alphabetic characters and is at least one character long. If you specify a value for OAuth Extensions field, ensure the input contains only printable ASCII characters or whitespace (including space, tab, carriage return, and newline), and the value is at least one character long. You don’t need to include the extension_ part, such as in extension_logicalCluster or extension_identityPoolId.

Login Refresh Window Factor

Number

Time for which the login refresh thread waits until a certain portion of the credential’s lifespan passes and attempts to refresh it. Acceptable values range from (0.5) 50% to (1.0) 100%, with a default of (0.8) 80% if unspecified.

0.8

Login Refresh Window Jitter

Number

The maximum random jitter added to the login refresh thread’s sleep time relative to the credential’s lifespan is between (0) 0% and (0.25) 25%, with a default of (0.05) 5% if unspecified.

0.05

Login Refresh Minimum Period

Number

The desired minimum time in seconds for the login refresh thread to wait before refreshing a credential. Legal values are between 0 and 900 (15 minutes).

60

Login Refresh Buffer

Number

The amount of buffer time in seconds before credential expiration to maintain when refreshing a credential. Legal values are between 0 and 3600 (1 hour). A default value of 300 (5 minutes) is used if no value is specified.

300

Use Persistent Connections

Boolean

Indicates whether to use persistent connections:

  • true

Mule uses persistent connections.

  • false

Mule closes the connection after the first request completes.

true

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput. By default, the number of connections is unlimited.

-1

Connection Idle Timeout

Number

The number of milliseconds that a connection can remain idle before it is closed. The value of this attribute is only used when persistent connections are enabled.

30000

Stream Response

Boolean

Whether received responses should be streamed, meaning processing continues as soon as all headers are parsed and the body streamed as it arrives. When enabled, the response MUST be eventually read since depending on the configured buffer size it may not fit into memory and processing will stop until space is available.

false

Response Buffer Size

Number

Size of the buffer that stores the HTTP response, in bytes. By default, the space is not limited.

-1

Client Socket Properties

Encapsulates the configurable properties for TCP client socket connections.

Proxy Config Params

Configures a proxy for outbound connections.

Follow Redirects

Boolean

Specifies whether to follow redirects or not.

false

Response Timeout

Number

Maximum time that the request element will block the execution of the flow waiting for the HTTP response.

60000

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy

### Producer SASL/SCRAM Connection Type Use Salted Challenge Response Authentication Mechanism (SCRAM) or SASL/SCRAM, a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username and password authentication like PLAIN. Apache Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/scram/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of 0 disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

EncryptionType

Enumeration, one of:

  • SHA256

  • SHA512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/PLAIN Connection Type Use SASL authenticated with a username and password. For more information, refer to the [Confluent](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html) documentation.
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Username

String

Username with which to login.

x

Password

String

Password with which to login.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

### Producer SASL/TOKEN Connection Type Use delegation tokens to authenticate to the Kafka cluster. > [!NOTE] Due to security reasons, a delegation token cannot be renewed if the initial > authentication uses a delegation token. For more information, refer to > [Delegation Token Support for Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka).
NameTypeDescriptionDefault ValueRequired

Bootstrap Server URLs

Array of String

List of servers to bootstrap the connection with the Kafka cluster. This can be a partial list of the available servers. For example, localhost:9071.

x

Batch size

Number

Producer attempts to batch records together into fewer requests whenever multiple records are sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. No attempt is made to batch records larger than this size. Requests sent to brokers contain multiple batches, one for each partition with the data that is available to send. A small batch size makes batching less common and can reduce throughput (a batch size of zero disables batching entirely). A very large batch size can result in more wasteful use of memory as a buffer of the specified batch size is always allocated in anticipation of additional records.

16

The batch size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Batch size field.

KB

Buffer size

Number

Total bytes of memory the producer uses to buffer records waiting to send to the server. If records are sent faster than they are delivered to the server, the producer blocks for max.block.ms, after which it throws an exception. This setting generally corresponds to the total memory the producer uses, but is not exact because the memory used by the producer is not all used for buffering. Some additional memory is used for compression (if compression is enabled), as well as for maintaining in-flight requests. The default value in the Apache Kafka documentation is 33554432 (32MB), but this should be capped to align with expected values for Mule instances in CloudHub (v0.1 core).

1

The buffer memory size unit of measure.

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Buffer size field.

KB

DNS Lookups

Enumeration, one of:

  • DEFAULT

  • USE_ALL_DNS_IPS

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

Controls how the client uses DNS lookups. If set to USE_ALL_DNS_IPS then, when the lookup returns multiple IP addresses for a hostname, they attempt to connect before failing the connection. Applies to both bootstrap and advertised servers. If the value is RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY each entry is resolved and expanded into a list of canonical names.

  • USE_ALL_DNS_IPS

    When the lookup returns multiple IP addresses for a hostname, a connection is attempted to all of the IP addresses before the connection fails. Applies to both bootstrap and advertised servers.

  • RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

    Each entry is resolved and expanded into a list of canonical names.

USE_ALL_DNS_IPS

Compression type

Enumeration, one of:

  • NONE

  • GZIP

  • SNAPPY

  • LZ4

  • ZSTD

Compression type for all data generated by the producer. The default is NONE (no compression). Compression works on full batches of data, so the efficacy of batching also impacts the compression ratio. More batching means better compression.

NONE

Connections maximum idle time

Number

Close idle connections after the specified time is reached.

540

Connections maximum idle time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Connections maximum idle time field.

SECONDS

Delivery Timeout

Number

Upper limit on the time to report success or failure after a call to send() returns. This limits the total time that a record is delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retrying send failures. The producer might report failure to send a record earlier than this configuration if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch that reached an earlier delivery expiration deadline. The value of this configuration must be greater than or equal to the sum of request.timeout.ms and linger.ms.

120

Delivery Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delivery Timeout field.

SECONDS

Enable idempotence

Boolean

When set to true, the producer ensures that exactly one copy of each message is written in the stream. If false, the producer retries due to broker failures and might write duplicates of the retried message in the stream. Enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5, retries to be greater than 0. and acks to be all. If these values are not explicitly set by the user, suitable values are chosen. If incompatible values are set, a ConnectionException is thrown.

false

Linger time

Number

Producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they are sent out. However, in some circumstances the client might want to reduce the number of requests, even under moderate load.

This setting accomplishes this by adding a small amount of artificial delay (rather than immediately sending out a record the producer waits for up to the given delay to allow other records to be sent so that the sends can be batched together). This is analogous to Nagle’s algorithm in TCP. This setting gives the upper bound on the delay for batching.

After the specified batch.size worth of records is received for a partition, it is sent immediately regardless of this setting, however if fewer than the specified number of bytes accumulated for this partition is received, the producer "lingers" for the specified time waiting for more records to arrive.

This setting defaults to 0 (no delay). Setting linger.ms=5, for example, has the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.

0

Linger Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Linger time field.

SECONDS

Maximum block time

Number

Controls for how long KafkaProducer.send() and KafkaProducer.partitionsFor() are blocked. These methods can be blocked either because the buffer is full or the metadata unavailable.Blocking in the user-supplied serializers or partitioner is not counted against this timeout.

60

Maximum block time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Maximum block time field.

SECONDS

Maximum in flight requests

Number

Maximum number of unacknowledged requests the client sends on a single connection before blocking. If the value is greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (if retries are enabled).

5

Maximum request size

Number

Maximum size of a request in bytes. This setting limits the number of record batches the producer sends in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size, which might be different from this.

1

Maximum request size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Maximum request size field.

MB

Producer Acknowledge Mode

Enumeration, one of:

  • NONE

  • LEADER_ONLY

  • ALL

Number of acknowledgments the producer requires the leader to receive before considering a request complete. This controls the durability of records that are sent.

NONE

Default receive buffer size

Number

Size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default is used. You can override this parameter at the source level.

64

Default receive buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default receive buffer size field. You can override this parameter at the source level.

KB

Retries amount

Number

Setting a value greater than 0 causes the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 potentially changes the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch might appear first. Additionally, produce requests will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires before successful acknowledgment. It is recommended that users leave this configuration unset and instead use delivery.timeout.ms to control retry behavior.

1

Retry Backoff Timeout

Number

Amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

100

Retry Backoff Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Retry Backoff Timeout field.

MILLISECONDS

Default send buffer size

Number

Size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default is used. You can override this parameter at the source level.

128

Default send buffer size unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit of measure for the Default send buffer size field. You can override this parameter at the source level.

KB

Default request timeout

Number

Maximum amount of time the client waits for the response to a request. If the response is not received before the timeout elapses, the client resends the request if necessary or fails the request if retries are exhausted. This value must be larger than replica.lag.time.max.ms (a broker configuration) to reduce the possibility of message duplication due to unnecessary producer retries.

30

Default request timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Default request timeout field.

SECONDS

Partitioner

Enumeration, one of:

  • NONE

  • DEFAULT

  • ROUND_ROBIN

  • UNIFORM_STICKY

Controls the partitioning strategy.

DEFAULT

Additional Properties

Object

Additional properties used to customize the Kafka connection.

For example:

  • For serialization, set Key to key.serializer or value.serializer and Value to com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroSerializer.

  • For deserialization, set Key to key.deserializer or value.deserializer and Value to io.confluent.kafka.serializers.KafkaAvroDeserializer.

TLS Configuration

Defines a TLS configuration, used by both clients and server sides to secure the communication for the Mule app. The connector automatically sets the security.protocol to use for the communication. The valid values are PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. If the broker configures SSL as the protocol, configure the keystore in the tls:context child element of the configuration and the connector will automatically use SSL (or SASL_SSL for SASL authentication) as the security.protocol.

Endpoint identification algorithm

String

Endpoint identification algorithm used by clients to validate the server hostname. The default value is an empty string, which means the endpoint identification algorithm is disabled. Clients, including client connections created by the broker for inter-broker communication, verify that the broker host name matches the host name in the brokers certificate.

Token ID

String

ID of the token.

x

Token HMAC

String

Token HMAC.

x

Encryption type

Enumeration, one of:

  • SCRAM_SHA_256

  • SCRAM_SHA_512

Encryption algorithm used by SCRAM.

x

Reconnection

Configures a reconnection strategy to use when a connector operation fails to connect to an external server.

## Consumer Sources - [Batch Message Listener](#batch-message-listener) - [Message Listener](#message-listener) ### Batch Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a list of messages to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Record

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) > [!NOTE] The **Batch Message Listener** source does not support configurable streaming strategies > because the source takes a batch of records as an input stream. The streaming strategy > configuration is non-repeatable-stream by default for the source. ### Message Listener `` This source supports the consumption of messages from a Kafka cluster, producing a single message to the flow.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Poll timeout

Number

Amount of time to block. Defines the total timeout for polling.

Poll timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Poll timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

Amount of parallel consumers.

Number

Number of parallel consumers.

1

Primary Node Only

Boolean

Determines whether to execute this source on only the primary node when running Mule instances in a cluster.

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) ## Consumer Operations - [Commit](#commit) - [Consume](#consume) - [Seek](#seek) ### Commit `` Commits the offsets associated to a message or batch of messages consumed in a **Message Listener**. This is a list or a single message consumed in the **Batch Message Listener** source.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumer commit key

String

Commit key of the last poll. This operation is valid only when used inside a flow that uses one of the sources (Batch Message Listener or Message Listener) which inserts this value as an attribute in the Mule event.

x

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_ACK\_MODE - KAFKA:RETRY\_EXHAUSTED - KAFKA:ALREADY\_COMMITTED - KAFKA:TIMEOUT - KAFKA:SESSION\_NOT\_FOUND - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Consume `` Receives messages from one or more Kafka topics. This operation works similarly to the **Message Listener** source, so all the operations that apply to that, apply to this operation as well. > [!NOTE] The **Consume** operation does not return the `consumerCommitKey`.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Consumption timeout

Number

Number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Consumption timeout field.

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Acknowledgement mode

Enumeration, one of:

  • AUTO

  • MANUAL

  • IMMEDIATE

  • DUPS_OK

Defines the way that the Kafka Broker instance is notified of the consumption of messages.

  • AUTO

    Commits messages automatically if the flow finishes successfully.

  • MANUAL

    The user must commit messages manually upon receipt by using the Commit operation.

  • IMMEDIATE

    Mule automatically commits the messages upon receipt before triggering the flow.

  • DUPS_OK

    Works in the same way as MANUAL, but the commits are asynchronous, which can lead to duplicate records.

IMMEDIATE

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Binary

Attributes Type

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:SECURITY - KAFKA:RETRY\_EXHAUSTED - KAFKA:ILLEGAL\_STATE - KAFKA:TIMEOUT - KAFKA:SERIALIZATION\_ERROR - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ### Seek `` Sets the current offset of the consumer for the given topic and partition to the provided offset value.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which the Seek operation is performed.

x

Partition

Number

Partition number that has its offset modified.

x

Offset

Number

Offset value to commit for the configured partition.

x

Operation Timeout

Number

Timeout for the operation to start executing.

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Operation Timeout field.

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Associated Configurations - [Consumer Configuration](#consumer-config) #### Throws - KAFKA:INVALID\_TOPIC - KAFKA:SECURITY - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INVALID\_OFFSET - KAFKA:INVALID\_INPUT - KAFKA:NOT\_FOUND - KAFKA:CONNECTIVITY ## Producer Operations - [Bulk Publish](#bulkPublish) - [Publish](#publish) ### Bulk Publish `` Publishes a bulk of messages to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Messages

Array of Kafka Message

List of messages to publish.

#[payload]

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Array of Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INVALID\_CONNECTION - KAFKA:AUTHENTICATION\_ERROR - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ### Publish `` Publishes a message to the specified Kafka topic, optionally specifying the partition, key, and message content for it. This operation supports transactions.
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Topic to publish to.

Partition

Number

(Optional) Topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

Type of joining action that operations can take for transactions.

JOIN_IF_POSSIBLE

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

Expression that evaluates the operation’s output. The outcome of the expression is stored in the Target Variable field.

#[payload]

Reconnection Strategy

Retry strategy in case of connectivity errors.

#### Output

Type

Kafka Message Metadata

#### Associated Configurations - [Producer Configuration](#producer-config) #### Throws - KAFKA:SECURITY - KAFKA:INVALID\_TOPIC\_PARTITION - KAFKA:RETRY\_EXHAUSTED - KAFKA:TIMEOUT - KAFKA:INVALID\_CONNECTION - KAFKA:AUTHENTICATION\_ERROR - KAFKA:INPUT\_TOO\_LARGE - KAFKA:CONNECTIVITY ## Object Types - [Consumer Context](#ConsumerContext) - [CRL File](#CrlFile) - [Custom OCSP Responder](#CustomOcspResponder) - [Expiration Policy](#ExpirationPolicy) - [Kafka Message](#KafkaMessage) - [Kafka Message Attributes](#KafkaMessageAttributes) - [Kafka Message Metadata](#KafkaMessageMetadata) - [Kafka Record Attributes](#KafkaRecordAttributes) - [Keystore](#KeyStore) - [Proxy Config Params](#ProxyConfigParams) - [Reconnect](#Reconnect) - [Reconnect Forever](#Reconnect-Forever) - [Reconnection](#Reconnection) - [Record](#Record) - [Ser Byte Array](#SerByteArray) - [Redelivery Policy](#RedeliveryPolicy) - [Repeatable File Store Stream](#RepeatableFileStoreStream) - [Repeatable In Memory Stream](#RepeatableInMemoryStream) - [Standard Revocation Check](#StandardRevocationCheck) - [Tcp Client Socket Params](#TcpClientSocketParams) - [TLS](#Tls) - [Topic Partition](#TopicPartition) - [Truststore](#TrustStore) ### Consumer Context Type for the consumer context.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

### CRL File Specifies the location of the certification revocation list (CRL) file.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the CRL file.

### Custom OCSP Responder Configures a custom OCSP responder for certification revocation checks.
FieldTypeDescriptionDefault ValueRequired

Url

String

URL of the OCSP responder.

Cert Alias

String

Alias of the signing certificate for the OCSP response. If specified, the alias must be in the truststore.

### Expiration Policy Configures an expiration policy strategy.
FieldTypeDescriptionDefault ValueRequired

Max Idle Time

Number

Configures the maximum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration.

Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Max Idle Time field.

### Kafka Message Represents a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Attributes

Attributes associated with the Kafka message.

Payload

Binary

Payload of the Kafka message, represented as an InputStream.

### Kafka Message Attributes Represents the attributes of a Kafka message.
FieldTypeDescriptionDefault ValueRequired

Headers

Object

Headers of the Kafka message, represented as a map in which the key is a string and the value is an InputStream.

Key

Binary

Key of the Kafka message, represented as an InputStream.

Partition

Number

Partition of the topic where the Kafka message is sent.

Topic

String

Topic to which the Kafka message belongs to.

### Kafka Message Metadata Metadata of the Kafka message.
FieldTypeDescriptionDefault ValueRequired

Offset

Number

Offset of the message.

Partition

Number

Partition of the message.

Serialized Key Size

Number

Serialized key size of the message.

Serialized Value Size

Number

Serialized value size of the message.

Timestamp

DateTime

Timestamp of the message.

Topic

String

Topic of the message.

### Kafka Record Attributes Attributes of the Kafka record.
FieldTypeDescriptionDefault ValueRequired

Consumer Commit Key

String

Consumer commit key.

Creation Timestamp

DateTime

Timestamp of record creation.

Headers

Object

Map of HTTP headers in the message.

Key

Binary

Keys of the record.

Leader Epoch

Number

Leader epoch of the record.

Log Append Timestamp

DateTime

Log append timestamp of the record.

Offset

Number

Offset of the record.

Partition

Number

Partition of the record.

Serialized Key Size

Number

Serialized key size of the record.

Serialized Value Size

Number

Serialized value size of the record.

Topic

String

Topic of the record.

### Keystore Configures the keystore for the TLS protocol. The keystore you generate contains a private key and a public certificate.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the keystore. Mule resolves the path relative to the current classpath and file system.

Type

String

Type of keystore.

Alias

String

Alias of the key to use when the keystore contains multiple private keys. By default, Mule uses the first key in the file.

Key Password

String

Password used to protect the private key.

Password

String

Password used to protect the keystore.

Algorithm

String

Encryption algorithm that the keystore uses.

### Proxy Config Params Configures the proxy configuration parameters.
FieldTypeDescriptionDefault ValueRequired

Host

String

Hostname or IP address of the proxy server.

Port

Number

Port of the proxy server.

Username

String

Username to authenticate against the proxy server.

Password

String

Password to authenticate against the proxy server.

### Reconnect Configures a standard reconnection strategy, which specifies how often to reconnect and how many reconnection attempts the connector source or operation can make.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

Count

Number

How many reconnection attempts the Mule app can make.

### Reconnect Forever Configures a forever reconnection strategy by which the connector source or operation attempts to reconnect at a specified frequency for as long as the Mule app runs.
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often to attempt to reconnect, in milliseconds.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

### Reconnection Configures a reconnection strategy for an operation.
FieldTypeDescriptionDefault ValueRequired

Fails Deployment

Boolean

What to do if, when an app is deployed, a connectivity test does not pass after exhausting the associated reconnection strategy:

  • true

    Allow the deployment to fail.

  • false

    Ignore the results of the connectivity test.

Reconnection Strategy

Reconnection strategy to use.

### Record Type for records.
FieldTypeDescriptionDefault ValueRequired

Attributes

Kafka record attributes.

Payload

Binary

Payload for the record.

Ser Byte Array

### Ser Byte Array
FieldTypeDescriptionDefault ValueRequired

Input Stream

Binary

### Redelivery Policy Configures the redelivery policy for executing requests that generate errors. You can add a redelivery policy to any source in a flow.
FieldTypeDescriptionDefault ValueRequired

Max Redelivery Count

Number

Maximum number of times that a redelivered request can be processed unsuccessfully before returning a REDELIVERY_EXHAUSTED error.

Use Secure Hash

Boolean

If true, Mule uses a secure hash algorithm to identify a redelivered message.

Message Digest Algorithm

String

Secure hashing algorithm to use if the Use Secure Hash field is true. If the payload of the message is a Java object, Mule ignores this value and returns the value that the payload’s hashCode() returned.

Id Expression

String

Defines one or more expressions to use to determine when a message has been redelivered. This property may only be set if Use Secure Hash is false.

Object Store

ObjectStore

Configures the object store that stores the redelivery counter for each message.

### Repeatable File Store Stream Configures the repeatable file-store streaming strategy by which Mule keeps a portion of the stream content in memory. If the stream content is larger than the configured buffer size, Mule backs up the buffer’s content to disk and then clears the memory.
FieldTypeDescriptionDefault ValueRequired

In Memory Size

Number

Maximum amount of memory that the stream can use for data. If the amount of memory exceeds this value, Mule buffers the content to disk. To optimize performance:

  • Configure a larger buffer size to avoid the number of times Mule needs to write the buffer on disk. This increases performance, but it also limits the number of concurrent requests your application can process, because it requires additional memory.

  • Configure a smaller buffer size to decrease memory load at the expense of response time.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the In Memory Size field.

### Repeatable In Memory Stream Configures the in-memory streaming strategy by which the request fails if the data exceeds the MAX buffer size. Always run performance tests to find the optimal buffer size for your specific use case.
FieldTypeDescriptionDefault ValueRequired

Initial Buffer Size

Number

Initial amount of memory to allocate to the data stream. If the streamed data exceeds this value, the buffer expands by Buffer Size Increment, with an upper limit of Max In Memory Size value.

Buffer Size Increment

Number

Amount by which the buffer size expands if it exceeds its initial size. Setting a value of 0 or lower specifies that the buffer can’t expand.

Max Buffer Size

Number

Maximum amount of memory to use. If more than that is used then a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value lower than or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the Initial Buffer Size, Buffer Size Increment, and Buffer Unit fields.

### Standard Revocation Check Configures standard revocation checks for TLS certificates.
FieldTypeDescriptionDefault ValueRequired

Only End Entities

Boolean

Which elements to verify in the certificate chain:

  • true

    Verify only the last element in the certificate chain.

  • false

    Verify all elements in the certificate chain.

Prefer Crls

Boolean

How to check certificate validity:

  • true

    Check the Certification Revocation List (CRL) for certificate validity.

  • false

    Use the Online Certificate Status Protocol (OCSP) to check certificate validity.

No Fallback

Boolean

Whether to use the secondary method to check certificate validity:

  • true

    Use the method that wasn’t specified in the Prefer Crls field (the secondary method) to check certificate validity.

  • false

    Do not use the secondary method to check certificate validity.

Soft Fail

Boolean

What to do if the revocation server can’t be reached or is busy:

  • true

    Avoid verification failure.

  • false

    Allow the verification to fail.

### Tcp Client Socket Params Configures the TCP client socket parameters.
FieldTypeDescriptionDefault ValueRequired

Send Buffer Size

Number

Size of the buffer (in bytes) used when sending data, set on the socket itself.

Receive Buffer Size

Number

Buffer size.

Client Timeout

Number

Sets the SO_TIMEOUT value on sockets. Indicates the amount of time (in milliseconds) that the socket waits in a blocking operation before failing. A value of 0 (the default) means waiting indefinitely.

0

Send Tcp No Delay

Boolean

If set, data is sent immediately rather than being collected for efficiency, prioritizing latency over network traffic reduction. Despite the socket default being false, this option defaults to true because reducing network traffic is generally not a major concern.

true

Linger

Number

Sets the SO_LINGER value in milliseconds. This determines how long the socket remains open to ensure all remaining data is transmitted correctly before closing.

Keep Alive

Boolean

Enables the SO_KEEPALIVE behavior on open sockets. This option periodically checks idle socket connections and closes them if they become unavailable. It is used by server sockets to control whether connections are kept alive before being recycled.

false

Connection Timeout

Number

How long the connector waits before timing out when establishing a connection to the remote service.

30000

### TLS Configures TLS to provide secure communications for the Mule app.
FieldTypeDescriptionDefault ValueRequired

Enabled Protocols

String

Comma-separated list of protocols enabled for this context.

Enabled Cipher Suites

String

Comma-separated list of cipher suites enabled for this context.

Trust Store

Configures the TLS truststore.

Key Store

Configures the TLS keystore.

Revocation Check

Configures a revocation checking mechanism.

### Topic Partition Type for topic partition.
FieldTypeDescriptionDefault ValueRequired

Topic

String

Topic name.

x

Partition

Number

Number of partitions.

x

### Truststore Configures the truststore for TLS.
FieldTypeDescriptionDefault ValueRequired

Path

String

Path to the truststore. Mule resolves the path relative to the current classpath and file system.

Password

String

Password used to protect the truststore.

Type

String

Type of truststore.

Algorithm

String

Encryption algorithm that the truststore uses.

Insecure

Boolean

If true, Mule stops performing certificate validations. Setting this to true can make connections vulnerable to attacks.

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Configure AWS IAM Authentication for Apache Kafka Connector > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Use AWS Identity and Access Management (IAM) authentication to connect Apache Kafka Connector to an Amazon MSK cluster. ## Configure the AWS IAM Driver in the POM File In the Mule app `pom.xml` file, configure the `aws-msk-iam-auth` dependency required for SASL/AWS IAM authentication. Add the dependency to the `additionalPluginDependencies` section for `mule-kafka-connector`. ```xml org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector software.amazon.msk aws-msk-iam-auth 2.3.4 ``` Use version `2.3.4` or later. > [!IMPORTANT] Don’t add `aws-msk-iam-auth` as a `` or ``. The driver > must resolve only through the connector’s `additionalPluginDependencies`. Adding it as a shared > library causes a `NoClassDefFoundError` for > `org.apache.kafka.common.security.auth.AuthenticateCallbackHandler` during deployment. If Anypoint > Studio rewrites this block into ``, change it back to > `additionalPluginDependencies`. ## Configure AWS IAM Authentication The SASL/AWS IAM connection supports two authentication modes, controlled by the **Try Default AWS Credentials Provider Chain** field. ### Configure Static Credentials with AssumeRole **Try Default AWS Credentials Provider Chain** is `false` by default. Provide **Access Key ID**, **Secret Access Key**, and **Role ARN**: ```xml ``` If the caller credentials are temporary, such as federated credentials, AWS SSO credentials, or credentials from a prior STS call, add **Session Token**. ```xml ``` ### Configure the Default Credentials Provider Chain Set **Try Default AWS Credentials Provider Chain** to `true` and leave **Access Key ID** and **Secret Access Key** empty. The identity is resolved from the runtime environment, such as system properties, environment variables, EKS IRSA, an ECS task role, an EC2 instance profile, or an `~/.aws` profile. For a roleless configuration, the resolved identity accesses the cluster directly: ```xml ``` To use AssumeRole, specify **Role ARN** for the resolved identity. This is common for CloudHub 2.0 private spaces and Runtime Fabric on EKS: ```xml ``` > [!NOTE] The default credentials provider chain works only on self-managed runtimes with an AWS > identity attached to the environment, such as Runtime Fabric on EKS, standalone Mule on EC2, or > CloudHub 2.0 private spaces. It doesn’t apply to CloudHub 1.0. Use static credentials with > AssumeRole instead. ## See Also - [Consumer SASL/AWS IAM Connection](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-reference#consumer-sasl-aws-iam) - [Producer SASL/AWS IAM Connection](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-reference#producer-sasl-aws-iam) - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-examples) --- product: Apache Kafka Connector version: 4.10 --- # Integrate Confluent Schema Registry with Kafka Connector - Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to configure the Confluent Schema Registry using Kafka Connector. Imagine you are developing a Mule 4 app that processes streaming data from various sources. To ensure the data conforms to defined schemas and to manage schema evolution effectively, you integrate Confluent Schema Registry with your Kafka Connector. This setup is crucial for developers who need to enforce data integrity and compatibility across different versions of data producers and consumers. The configuration consists of two parts: - Configuring the Confluent Schema Registry in the Kafka Connector connection using the **Additional Properties** field for the **Consumer** operation. - Configuring the Kafka Avro dependency in the POM file of the Mule app ## Configure the Confluent Schema Registry in the Kafka Connector Connection Use the **Additional Properties** field to configure the Confluent Schema Registry. ### Configure the Consumer Operation To configure the **Consumer** operation in Anypoint Studio: 1. Open your Mule project in Anypoint Studio and navigate to the **Global Elements** tab. 2. Click **Create**. 3. In the search box, type `kafka`. 4. Select **Apache Kafka Consumer configuration**. 5. Click **OK**. 6. In the configuration screen under connection, select **Advanced**. 7. Fill in all the required connection parameters. 8. For **Additional properties**, select **Edit inline**. 9. Click the plus sign to add a new additional property. 10. Set **Key** to `key.deserializer` and **Value** to `io.confluent.kafka.serializers.KafkaAvroDeserializer`. 11. Repeat step 9 to add all the required Confluent Schema Registry properties. In the **Configuration XML** window, check the complete confluent schema registry configuration under the `` tags: ```xml ``` ## Configure the Kafka Avro dependency in the POM File In the POM file of the Mule app, configure the Kafka Avro dependency that contains the class to use to serialize and deserialize. You must update the ``, ``, and `` sections of the POM file. ### Configure the Section In the `` section, set `kafka-avro-serializer`: ```xml io.confluent kafka-avro-serializer 7.4.0 ``` ### Configure the Section In the `` section, set `kafka-avro-serializer` as a ``: ```xml org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 io.confluent kafka-avro-serializer mule-application ``` Then, set the `avro-maven-plugin` plugin. This dependency from the `org.apache.avro` group is a plugin for Apache Maven that facilitates handling Avro schema files in Java projects. The plugin automatically generates Java classes from Avro schemas (.avsc), protocols (.avpr), and IDL files (.avdl). This process simplifies development by eliminating the need to manually write Java classes that represent Avro schemas, ensuring that the Java code is always synchronized with the Avro schemas. The `` specifices the folder where the Avro schemas live, for example `${project.basedir}/PATH_TO_SCHEMAS`. ```xml org.apache.avro avro-maven-plugin 1.10.2 generate-sources schema ${project.basedir}/PATH_TO_SCHEMAS ``` ### Configure the Section In the `` section, configure the repository of the dependency: ```xml confluent https://packages.confluent.io/maven/ ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-examples) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.11 --- # Integrate Confluent Schema Registry with Kafka Connector - Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to configure the Confluent Schema Registry using Kafka Connector. Imagine you are developing a Mule 4 app that processes streaming data from various sources. To ensure the data conforms to defined schemas and to manage schema evolution effectively, you integrate Confluent Schema Registry with your Kafka Connector. This setup is crucial for developers who need to enforce data integrity and compatibility across different versions of data producers and consumers. The configuration consists of two parts: - Configuring the Confluent Schema Registry in the Kafka Connector connection using the **Additional Properties** field for the **Consumer** operation. - Configuring the Kafka Avro dependency in the POM file of the Mule app ## Configure the Confluent Schema Registry in the Kafka Connector Connection Use the **Additional Properties** field to configure the Confluent Schema Registry. ### Configure the Consumer Operation To configure the **Consumer** operation in Anypoint Studio: 1. Open your Mule project in Anypoint Studio and navigate to the **Global Elements** tab. 2. Click **Create**. 3. In the search box, type `kafka`. 4. Select **Apache Kafka Consumer configuration**. 5. Click **OK**. 6. In the configuration screen under connection, select **Advanced**. 7. Fill in all the required connection parameters. 8. For **Additional properties**, select **Edit inline**. 9. Click the plus sign to add a new additional property. 10. Set **Key** to `key.deserializer` and **Value** to `io.confluent.kafka.serializers.KafkaAvroDeserializer`. 11. Repeat step 9 to add all the required Confluent Schema Registry properties. In the **Configuration XML** window, check the complete confluent schema registry configuration under the `` tags: ```xml ``` ## Configure the Kafka Avro dependency in the POM File In the POM file of the Mule app, configure the Kafka Avro dependency that contains the class to use to serialize and deserialize. You must update the ``, ``, and `` sections of the POM file. ### Configure the Section In the `` section, set `kafka-avro-serializer`: ```xml io.confluent kafka-avro-serializer 7.4.0 ``` ### Configure the Section In the `` section, set `kafka-avro-serializer` as a ``: ```xml org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 io.confluent kafka-avro-serializer mule-application ``` Then, set the `avro-maven-plugin` plugin. This dependency from the `org.apache.avro` group is a plugin for Apache Maven that facilitates handling Avro schema files in Java projects. The plugin automatically generates Java classes from Avro schemas (.avsc), protocols (.avpr), and IDL files (.avdl). This process simplifies development by eliminating the need to manually write Java classes that represent Avro schemas, ensuring that the Java code is always synchronized with the Avro schemas. The `` specifices the folder where the Avro schemas live, for example `${project.basedir}/PATH_TO_SCHEMAS`. ```xml org.apache.avro avro-maven-plugin 1.10.2 generate-sources schema ${project.basedir}/PATH_TO_SCHEMAS ``` ### Configure the Section In the `` section, configure the repository of the dependency: ```xml confluent https://packages.confluent.io/maven/ ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-examples) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.12 --- # Integrate Confluent Schema Registry with Kafka Connector - Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to configure the Confluent Schema Registry using Kafka Connector. Imagine you are developing a Mule 4 app that processes streaming data from various sources. To ensure the data conforms to defined schemas and to manage schema evolution effectively, you integrate Confluent Schema Registry with your Kafka Connector. This setup is crucial for developers who need to enforce data integrity and compatibility across different versions of data producers and consumers. The configuration consists of two parts: - Configuring the Confluent Schema Registry in the Kafka Connector connection using the **Additional Properties** field for the **Consumer** operation. - Configuring the Kafka Avro dependency in the POM file of the Mule app ## Configure the Confluent Schema Registry in the Kafka Connector Connection Use the **Additional Properties** field to configure the Confluent Schema Registry. ### Configure the Consumer Operation To configure the **Consumer** operation in Anypoint Studio: 1. Open your Mule project in Anypoint Studio and navigate to the **Global Elements** tab. 2. Click **Create**. 3. In the search box, type `kafka`. 4. Select **Apache Kafka Consumer configuration**. 5. Click **OK**. 6. In the configuration screen under connection, select **Advanced**. 7. Fill in all the required connection parameters. 8. For **Additional properties**, select **Edit inline**. 9. Click the plus sign to add a new additional property. 10. Set **Key** to `key.deserializer` and **Value** to `com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroDeserializer`. 11. Repeat step 9 to add all the required Confluent Schema Registry properties. Optionally, you can configure these system properties `maxCacheSize`, `maxCacheTTL` for the key `mule.kafka.muleKafkaAvroDeserializer`:
PropertyDescriptionDefault

mule.kafka.muleKafkaAvroDeserializer.maxCacheSize

The maximum number of schemas to store in cache.

200

mule.kafka.muleKafkaAvroDeserializer.maxCacheTTL

The time-to-live (TTL) in minutes that a schema remains in cache when not being used.

30 minutes

In the **Configuration XML** window, check the complete confluent schema registry configuration under the `` tags: ```xml ``` ## Configure the Kafka Avro dependency in the POM File In the POM file of the Mule app, configure the Kafka Avro dependency that contains the class to use to serialize and deserialize. You must update the ``, ``, and `` sections of the POM file. ### Configure the Section In the `` section, set `kafka-avro-serializer`: ```xml io.confluent kafka-avro-serializer 7.4.0 ``` ### Configure the Section In the `` section, set `kafka-avro-serializer` as a ``: ```xml org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 io.confluent kafka-avro-serializer mule-application ``` Then, set the `avro-maven-plugin` plugin. This dependency from the `org.apache.avro` group is a plugin for Apache Maven that facilitates handling Avro schema files in Java projects. The plugin automatically generates Java classes from Avro schemas (.avsc), protocols (.avpr), and IDL files (.avdl). This process simplifies development by eliminating the need to manually write Java classes that represent Avro schemas, ensuring that the Java code is always synchronized with the Avro schemas. The `` specifices the folder where the Avro schemas live, for example `${project.basedir}/PATH_TO_SCHEMAS`. ```xml org.apache.avro avro-maven-plugin 1.10.2 generate-sources schema ${project.basedir}/PATH_TO_SCHEMAS ``` ### Configure the Section In the `` section, configure the repository of the dependency: ```xml confluent https://packages.confluent.io/maven/ ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-examples) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.13 --- # Integrate Confluent Schema Registry with Kafka Connector - Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to configure the Confluent Schema Registry using Kafka Connector. Imagine you are developing a Mule 4 app that processes streaming data from various sources. To ensure the data conforms to defined schemas and to manage schema evolution effectively, you integrate Confluent Schema Registry with your Kafka Connector. This setup is crucial for developers who need to enforce data integrity and compatibility across different versions of data producers and consumers. The configuration consists of two parts: - Configuring the Confluent Schema Registry in the Kafka Connector connection using the **Additional Properties** field for the **Consumer** operation. - Configuring the Kafka Avro dependency in the POM file of the Mule app ## Configure the Confluent Schema Registry in the Kafka Connector Connection Use the **Additional Properties** field to configure the Confluent Schema Registry. ### Configure the Consumer Operation To configure the **Consumer** operation in Anypoint Studio: 1. Open your Mule project in Anypoint Studio and navigate to the **Global Elements** tab. 2. Click **Create**. 3. In the search box, type `kafka`. 4. Select **Apache Kafka Consumer configuration**. 5. Click **OK**. 6. In the configuration screen under connection, select **Advanced**. 7. Fill in all the required connection parameters. 8. For **Additional properties**, select **Edit inline**. 9. Click the plus sign to add a new additional property. 10. Set **Key** to `key.deserializer` and **Value** to `com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroDeserializer`. 11. Repeat step 9 to add all the required Confluent Schema Registry properties. Optionally, you can configure these system properties `maxCacheSize`, `maxCacheTTL` for the key `mule.kafka.muleKafkaAvroDeserializer`:
PropertyDescriptionDefault

mule.kafka.muleKafkaAvroDeserializer.maxCacheSize

The maximum number of schemas to store in cache.

200

mule.kafka.muleKafkaAvroDeserializer.maxCacheTTL

The time-to-live (TTL) in minutes that a schema remains in cache when not being used.

30 minutes

In the **Configuration XML** window, check the complete confluent schema registry configuration under the `` tags: ```xml ``` ## Configure the Kafka Avro dependency in the POM File In the POM file of the Mule app, configure the Kafka Avro dependency that contains the class to use to serialize and deserialize. You must update the ``, ``, and `` sections of the POM file. ### Configure the Section In the `` section, set `kafka-avro-serializer`: ```xml io.confluent kafka-avro-serializer 7.4.0 ``` ### Configure the Section In the `` section, set `kafka-avro-serializer` as a ``: ```xml org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 io.confluent kafka-avro-serializer mule-application ``` Then, set the `avro-maven-plugin` plugin. This dependency from the `org.apache.avro` group is a plugin for Apache Maven that facilitates handling Avro schema files in Java projects. The plugin automatically generates Java classes from Avro schemas (.avsc), protocols (.avpr), and IDL files (.avdl). This process simplifies development by eliminating the need to manually write Java classes that represent Avro schemas, ensuring that the Java code is always synchronized with the Avro schemas. The `` specifices the folder where the Avro schemas live, for example `${project.basedir}/PATH_TO_SCHEMAS`. ```xml org.apache.avro avro-maven-plugin 1.10.2 generate-sources schema ${project.basedir}/PATH_TO_SCHEMAS ``` ### Configure the Section In the `` section, configure the repository of the dependency: ```xml confluent https://packages.confluent.io/maven/ ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-examples) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.14 --- # Integrate Confluent Schema Registry with Kafka Connector - Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to configure the Confluent Schema Registry using Kafka Connector. Imagine you are developing a Mule 4 app that processes streaming data from various sources. To ensure the data conforms to defined schemas and to manage schema evolution effectively, you integrate Confluent Schema Registry with your Kafka Connector. This setup is crucial for developers who need to enforce data integrity and compatibility across different versions of data producers and consumers. The configuration consists of two parts: - Configuring the Confluent Schema Registry in the Kafka Connector connection using the **Additional Properties** field for the **Consumer** operation. - Configuring the Kafka Avro dependency in the POM file of the Mule app ## Configure the Confluent Schema Registry in the Kafka Connector Connection Use the **Additional Properties** field to configure the Confluent Schema Registry. ### Configure the Consumer Operation To configure the **Consumer** operation in Anypoint Studio: 1. Open your Mule project in Anypoint Studio and navigate to the **Global Elements** tab. 2. Click **Create**. 3. In the search box, type `kafka`. 4. Select **Apache Kafka Consumer configuration**. 5. Click **OK**. 6. In the configuration screen under connection, select **Advanced**. 7. Fill in all the required connection parameters. 8. For **Additional properties**, select **Edit inline**. 9. Click the plus sign to add a new additional property. 10. Set **Key** to `key.deserializer` and **Value** to `com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroDeserializer`. 11. Repeat step 9 to add all the required Confluent Schema Registry properties. Optionally, you can configure these system properties `maxCacheSize`, `maxCacheTTL` for the key `mule.kafka.muleKafkaAvroDeserializer`:
PropertyDescriptionDefault

mule.kafka.muleKafkaAvroDeserializer.maxCacheSize

The maximum number of schemas to store in cache.

200

mule.kafka.muleKafkaAvroDeserializer.maxCacheTTL

The time-to-live (TTL) in minutes that a schema remains in cache when not being used.

30 minutes

In the **Configuration XML** window, check the complete confluent schema registry configuration under the `` tags: ```xml ``` ## Configure the Kafka Avro dependency in the POM File In the POM file of the Mule app, configure the Kafka Avro dependency that contains the class to use to serialize and deserialize. You must update the ``, ``, and `` sections of the POM file. ### Configure the Section In the `` section, set `kafka-avro-serializer`: ```xml io.confluent kafka-avro-serializer 7.4.0 ``` ### Configure the Section In the `` section, set `kafka-avro-serializer` as a ``: ```xml org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 io.confluent kafka-avro-serializer mule-application ``` Then, set the `avro-maven-plugin` plugin. This dependency from the `org.apache.avro` group is a plugin for Apache Maven that facilitates handling Avro schema files in Java projects. The plugin automatically generates Java classes from Avro schemas (.avsc), protocols (.avpr), and IDL files (.avdl). This process simplifies development by eliminating the need to manually write Java classes that represent Avro schemas, ensuring that the Java code is always synchronized with the Avro schemas. The `` specifices the folder where the Avro schemas live, for example `${project.basedir}/PATH_TO_SCHEMAS`. ```xml org.apache.avro avro-maven-plugin 1.10.2 generate-sources schema ${project.basedir}/PATH_TO_SCHEMAS ``` ### Configure the Section In the `` section, configure the repository of the dependency: ```xml confluent https://packages.confluent.io/maven/ ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-examples) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Integrate Confluent Schema Registry with Kafka Connector - Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to configure the Confluent Schema Registry using Kafka Connector. Imagine you are developing a Mule 4 app that processes streaming data from various sources. To ensure the data conforms to defined schemas and to manage schema evolution effectively, you integrate Confluent Schema Registry with your Kafka Connector. This setup is crucial for developers who need to enforce data integrity and compatibility across different versions of data producers and consumers. The configuration consists of two parts: - Configuring the Confluent Schema Registry in the Kafka Connector connection using the **Additional Properties** field for the **Consumer** operation. - Configuring the Kafka Avro dependency in the POM file of the Mule app ## Configure the Confluent Schema Registry in the Kafka Connector Connection Use the **Additional Properties** field to configure the Confluent Schema Registry. ### Configure the Consumer Operation To configure the **Consumer** operation in Anypoint Studio: 1. Open your Mule project in Anypoint Studio and navigate to the **Global Elements** tab. 2. Click **Create**. 3. In the search box, type `kafka`. 4. Select **Apache Kafka Consumer configuration**. 5. Click **OK**. 6. In the configuration screen under connection, select **Advanced**. 7. Fill in all the required connection parameters. 8. For **Additional properties**, select **Edit inline**. 9. Click the plus sign to add a new additional property. 10. Set **Key** to `key.deserializer` and **Value** to `com.mulesoft.connectors.kafka.internal.model.serializer.MuleKafkaAvroDeserializer`. 11. Repeat step 9 to add all the required Confluent Schema Registry properties. Optionally, you can configure the system properties `maxCacheSize`, `maxCacheTTL`, or `useLogicalTypeAwareRenderer` for the key `mule.kafka.muleKafkaAvroDeserializer`:
PropertyDescriptionDefault

mule.kafka.muleKafkaAvroDeserializer.maxCacheSize

The maximum number of schemas to store in cache.

200

mule.kafka.muleKafkaAvroDeserializer.maxCacheTTL

The time-to-live (TTL) in minutes that a schema remains in cache when not being used.

30 minutes

mule.kafka.muleKafkaAvroDeserializer.useLogicalTypeAwareRenderer

When set to true, the deserializer represents each Avro logical type field according to its semantic value rather than its physical wire representation. For example, a decimal field is represented as an exact numeric string such as "9.99" instead of as raw bytes and a scale, and date, time, or timestamp fields are represented as ISO-8601 strings. Set this to false to maintain the legacy physical type representation and ensure backward compatibility.

false

In the **Configuration XML** window, check the complete confluent schema registry configuration under the `` tags: ```xml ``` ## Configure the Kafka Avro dependency in the POM File In the POM file of the Mule app, configure the Kafka Avro dependency that contains the class to use to serialize and deserialize. You must update the ``, ``, and `` sections of the POM file. ### Configure the Section In the `` section, set `kafka-avro-serializer`: ```xml io.confluent kafka-avro-serializer 7.4.0 ``` ### Configure the Section In the `` section, set `kafka-avro-serializer` as a ``: ```xml org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true com.mulesoft.connectors mule-kafka-connector io.confluent kafka-avro-serializer 7.6.1 io.confluent kafka-avro-serializer mule-application ``` Then, set the `avro-maven-plugin` plugin. This dependency from the `org.apache.avro` group is a plugin for Apache Maven that facilitates handling Avro schema files in Java projects. The plugin automatically generates Java classes from Avro schemas (.avsc), protocols (.avpr), and IDL files (.avdl). This process simplifies development by eliminating the need to manually write Java classes that represent Avro schemas, ensuring that the Java code is always synchronized with the Avro schemas. The `` specifices the folder where the Avro schemas live, for example `${project.basedir}/PATH_TO_SCHEMAS`. ```xml org.apache.avro avro-maven-plugin 1.10.2 generate-sources schema ${project.basedir}/PATH_TO_SCHEMAS ``` ### Configure the Section In the `` section, configure the repository of the dependency: ```xml confluent https://packages.confluent.io/maven/ ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-examples) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.10 --- # Publish and Then Consume a Topic with Kafka Connector > For the full documentation index, see: https://docs.mulesoft.com/llms.txt This example shows how to use two Apache Kafka Connector operations, **Publish** and **Consume**, to publish a message to Apache Kafka and then retrieve it. The example contains two flows: - The first flow is the Producer flow, which publishes a message to Apache Kafka. - The second flow is the Consumer flow, which consumes a message from Apache Kafka. ![Apache Kafka Connector Producer flow in Studio](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-producer-studio-flow.png) ![Apache Kafka Connector Consumer flow in Studio](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-consumer-studio-flow.png) ## Configure the First Flow Configure the first flow that publishes a message to Apache Kafka. Creating the first flow involves configuring a **Listener** component, a **Logger** component, a **Publish** operation, and a **Set Payload** transformer. ### Configure the HTTP Listener Configure the **Listener** component to initiate a Mule flow when a call is made to the `/pushMessage` path: 1. Create a new Mule project in Studio. 2. From the **Mule Palette** view, select **HTTP** and drag the **Listener** component to the canvas. 3. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 4. Accept the defaults. 5. In the properties window, set the **Path** field value to `/pushMessage`. ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

#["Message: '" payload.message "' is going to be published to topic: '" payload.topic "'."]

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-logger-config-publish.png) ### Add the Publish Operation Add the **Publish** operation to publish a message to the specified Kafka topic: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Publish** operation to the right of **Logger**. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector operation.

Connector configuration

Global configuration you just created.

Topic

\#[payload.topic]

Key

#[now()]

Message

payload

The following image shows an example of the **Publish** configuration in the properties window: ![Publish properties window configuration](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-publish.png) ### Add the Set Payload Transformer Add the **Set Payload** transformer to push the response builder: 1. From the **Mule Palette** view, select **Core** and drag the **Set Payload** transformer to the right of **Publish**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the transformer, such as Push response builder.

Value

Message successfully sent to Apache Kafka topic.

The following image shows an example of the **Set Payload** configuration in the properties window: ![Set Payload properties window configuration](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-set-payload.png) ## Configure the Second Flow Configure the second flow that consumes a message from Apache Kafka. Creating the second flow involves configuring a **Message listener** source and a **Logger** component. ### Add the Message Listener Source Add the **Message listener** source to consume a message endpoint: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Message listener** source to the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish.** 3. In the **Group ID** field, enter `${consumer.groupId}`. 4. In the **Topic Subscription Patterns** field, select **Edit inline** and then click the green plus sign. 5. Enter the value `${topic1}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector source.

Connector configuration

Global configuration you just created.

The following image shows an example of the **Message listener** configuration in the properties window: ![Message listener properties window configuration](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-message-listener-config.png) ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Message listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

"New message arrived: " payload ", key:" attributes.key ", partition:" attributes.partition ", offset:" ++ attributes.offset

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-logger-config-consume.png) ## XML for This Example Paste this code into the Studio XML editor to quickly load the flow for this example into your Mule app: ```xml ``` ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.11 --- # Publish and Then Consume a Topic with Kafka Connector > For the full documentation index, see: https://docs.mulesoft.com/llms.txt This example shows how to use two Apache Kafka Connector operations, **Publish** and **Consume**, to publish a message to Apache Kafka and then retrieve it. The example contains two flows: - The first flow is the Producer flow, which publishes a message to Apache Kafka. - The second flow is the Consumer flow, which consumes a message from Apache Kafka. ![Apache Kafka Connector Producer flow in Studio](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-producer-studio-flow.png) ![Apache Kafka Connector Consumer flow in Studio](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-consumer-studio-flow.png) ## Configure the First Flow Configure the first flow that publishes a message to Apache Kafka. Creating the first flow involves configuring a **Listener** component, a **Logger** component, a **Publish** operation, and a **Set Payload** transformer. ### Configure the HTTP Listener Configure the **Listener** component to initiate a Mule flow when a call is made to the `/pushMessage` path: 1. Create a new Mule project in Studio. 2. From the **Mule Palette** view, select **HTTP** and drag the **Listener** component to the canvas. 3. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 4. Accept the defaults. 5. In the properties window, set the **Path** field value to `/pushMessage`. ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

#["Message: '" payload.message "' is going to be published to topic: '" payload.topic "'."]

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-logger-config-publish.png) ### Add the Publish Operation Add the **Publish** operation to publish a message to the specified Kafka topic: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Publish** operation to the right of **Logger**. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector operation.

Connector configuration

Global configuration you just created.

Topic

\#[payload.topic]

Key

#[now()]

Message

payload

The following image shows an example of the **Publish** configuration in the properties window: ![Publish properties window configuration](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-publish.png) ### Add the Set Payload Transformer Add the **Set Payload** transformer to push the response builder: 1. From the **Mule Palette** view, select **Core** and drag the **Set Payload** transformer to the right of **Publish**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the transformer, such as Push response builder.

Value

Message successfully sent to Apache Kafka topic.

The following image shows an example of the **Set Payload** configuration in the properties window: ![Set Payload properties window configuration](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-set-payload.png) ## Configure the Second Flow Configure the second flow that consumes a message from Apache Kafka. Creating the second flow involves configuring a **Message listener** source and a **Logger** component. ### Add the Message Listener Source Add the **Message listener** source to consume a message endpoint: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Message listener** source to the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish.** 3. In the **Group ID** field, enter `${consumer.groupId}`. 4. In the **Topic Subscription Patterns** field, select **Edit inline** and then click the green plus sign. 5. Enter the value `${topic1}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector source.

Connector configuration

Global configuration you just created.

The following image shows an example of the **Message listener** configuration in the properties window: ![Message listener properties window configuration](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-message-listener-config.png) ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Message listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

"New message arrived: " payload ", key:" attributes.key ", partition:" attributes.partition ", offset:" ++ attributes.offset

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-logger-config-consume.png) ## XML for This Example Paste this code into the Studio XML editor to quickly load the flow for this example into your Mule app: ```xml ``` ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.12 --- # Publish and Then Consume a Topic with Kafka Connector > For the full documentation index, see: https://docs.mulesoft.com/llms.txt This example shows how to use two Apache Kafka Connector operations, **Publish** and **Consume**, to publish a message to Apache Kafka and then retrieve it. The example contains two flows: - The first flow is the Producer flow, which publishes a message to Apache Kafka. - The second flow is the Consumer flow, which consumes a message from Apache Kafka. ![Apache Kafka Connector Producer flow in Studio](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-producer-studio-flow.png) ![Apache Kafka Connector Consumer flow in Studio](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-consumer-studio-flow.png) ## Configure the First Flow Configure the first flow that publishes a message to Apache Kafka. Creating the first flow involves configuring a **Listener** component, a **Logger** component, a **Publish** operation, and a **Set Payload** transformer. ### Configure the HTTP Listener Configure the **Listener** component to initiate a Mule flow when a call is made to the `/pushMessage` path: 1. Create a new Mule project in Studio. 2. From the **Mule Palette** view, select **HTTP** and drag the **Listener** component to the canvas. 3. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 4. Accept the defaults. 5. In the properties window, set the **Path** field value to `/pushMessage`. ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

#["Message: '" payload.message "' is going to be published to topic: '" payload.topic "'."]

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-logger-config-publish.png) ### Add the Publish Operation Add the **Publish** operation to publish a message to the specified Kafka topic: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Publish** operation to the right of **Logger**. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector operation.

Connector configuration

Global configuration you just created.

Topic

\#[payload.topic]

Key

#[now()]

Message

payload

The following image shows an example of the **Publish** configuration in the properties window: ![Publish properties window configuration](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-publish.png) ### Add the Set Payload Transformer Add the **Set Payload** transformer to push the response builder: 1. From the **Mule Palette** view, select **Core** and drag the **Set Payload** transformer to the right of **Publish**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the transformer, such as Push response builder.

Value

Message successfully sent to Apache Kafka topic.

The following image shows an example of the **Set Payload** configuration in the properties window: ![Set Payload properties window configuration](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-set-payload.png) ## Configure the Second Flow Configure the second flow that consumes a message from Apache Kafka. Creating the second flow involves configuring a **Message listener** source and a **Logger** component. ### Add the Message Listener Source Add the **Message listener** source to consume a message endpoint: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Message listener** source to the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish.** 3. In the **Group ID** field, enter `${consumer.groupId}`. 4. In the **Topic Subscription Patterns** field, select **Edit inline** and then click the green plus sign. 5. Enter the value `${topic1}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector source.

Connector configuration

Global configuration you just created.

The following image shows an example of the **Message listener** configuration in the properties window: ![Message listener properties window configuration](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-message-listener-config.png) ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Message listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

"New message arrived: " payload ", key:" attributes.key ", partition:" attributes.partition ", offset:" ++ attributes.offset

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-logger-config-consume.png) ## XML for This Example Paste this code into the Studio XML editor to quickly load the flow for this example into your Mule app: ```xml ``` ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.13 --- # Publish and Then Consume a Topic with Kafka Connector > For the full documentation index, see: https://docs.mulesoft.com/llms.txt This example shows how to use two Apache Kafka Connector operations, **Publish** and **Consume**, to publish a message to Apache Kafka and then retrieve it. The example contains two flows: - The first flow is the Producer flow, which publishes a message to Apache Kafka. - The second flow is the Consumer flow, which consumes a message from Apache Kafka. ![Apache Kafka Connector Producer flow in Studio](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-producer-studio-flow.png) ![Apache Kafka Connector Consumer flow in Studio](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-consumer-studio-flow.png) ## Configure the First Flow Configure the first flow that publishes a message to Apache Kafka. Creating the first flow involves configuring a **Listener** component, a **Logger** component, a **Publish** operation, and a **Set Payload** transformer. ### Configure the HTTP Listener Configure the **Listener** component to initiate a Mule flow when a call is made to the `/pushMessage` path: 1. Create a new Mule project in Studio. 2. From the **Mule Palette** view, select **HTTP** and drag the **Listener** component to the canvas. 3. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 4. Accept the defaults. 5. In the properties window, set the **Path** field value to `/pushMessage`. ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

#["Message: '" payload.message "' is going to be published to topic: '" payload.topic "'."]

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-logger-config-publish.png) ### Add the Publish Operation Add the **Publish** operation to publish a message to the specified Kafka topic: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Publish** operation to the right of **Logger**. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector operation.

Connector configuration

Global configuration you just created.

Topic

\#[payload.topic]

Key

#[now()]

Message

payload

The following image shows an example of the **Publish** configuration in the properties window: ![Publish properties window configuration](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-publish.png) ### Add the Set Payload Transformer Add the **Set Payload** transformer to push the response builder: 1. From the **Mule Palette** view, select **Core** and drag the **Set Payload** transformer to the right of **Publish**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the transformer, such as Push response builder.

Value

Message successfully sent to Apache Kafka topic.

The following image shows an example of the **Set Payload** configuration in the properties window: ![Set Payload properties window configuration](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-set-payload.png) ## Configure the Second Flow Configure the second flow that consumes a message from Apache Kafka. Creating the second flow involves configuring a **Message listener** source and a **Logger** component. ### Add the Message Listener Source Add the **Message listener** source to consume a message endpoint: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Message listener** source to the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish.** 3. In the **Group ID** field, enter `${consumer.groupId}`. 4. In the **Topic Subscription Patterns** field, select **Edit inline** and then click the green plus sign. 5. Enter the value `${topic1}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector source.

Connector configuration

Global configuration you just created.

The following image shows an example of the **Message listener** configuration in the properties window: ![Message listener properties window configuration](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-message-listener-config.png) ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Message listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

"New message arrived: " payload ", key:" attributes.key ", partition:" attributes.partition ", offset:" ++ attributes.offset

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-logger-config-consume.png) ## XML for This Example Paste this code into the Studio XML editor to quickly load the flow for this example into your Mule app: ```xml ``` ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.14 --- # Publish and Then Consume a Topic with Kafka Connector > For the full documentation index, see: https://docs.mulesoft.com/llms.txt This example shows how to use two Apache Kafka Connector operations, **Publish** and **Consume**, to publish a message to Apache Kafka and then retrieve it. The example contains two flows: - The first flow is the Producer flow, which publishes a message to Apache Kafka. - The second flow is the Consumer flow, which consumes a message from Apache Kafka. ![Apache Kafka Connector Producer flow in Studio](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-producer-studio-flow.png) ![Apache Kafka Connector Consumer flow in Studio](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-consumer-studio-flow.png) ## Configure the First Flow Configure the first flow that publishes a message to Apache Kafka. Creating the first flow involves configuring a **Listener** component, a **Logger** component, a **Publish** operation, and a **Set Payload** transformer. ### Configure the HTTP Listener Configure the **Listener** component to initiate a Mule flow when a call is made to the `/pushMessage` path: 1. Create a new Mule project in Studio. 2. From the **Mule Palette** view, select **HTTP** and drag the **Listener** component to the canvas. 3. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 4. Accept the defaults. 5. In the properties window, set the **Path** field value to `/pushMessage`. ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

#["Message: '" payload.message "' is going to be published to topic: '" payload.topic "'."]

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-logger-config-publish.png) ### Add the Publish Operation Add the **Publish** operation to publish a message to the specified Kafka topic: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Publish** operation to the right of **Logger**. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector operation.

Connector configuration

Global configuration you just created.

Topic

\#[payload.topic]

Key

#[now()]

Message

payload

The following image shows an example of the **Publish** configuration in the properties window: ![Publish properties window configuration](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-publish.png) ### Add the Set Payload Transformer Add the **Set Payload** transformer to push the response builder: 1. From the **Mule Palette** view, select **Core** and drag the **Set Payload** transformer to the right of **Publish**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the transformer, such as Push response builder.

Value

Message successfully sent to Apache Kafka topic.

The following image shows an example of the **Set Payload** configuration in the properties window: ![Set Payload properties window configuration](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-set-payload.png) ## Configure the Second Flow Configure the second flow that consumes a message from Apache Kafka. Creating the second flow involves configuring a **Message listener** source and a **Logger** component. ### Add the Message Listener Source Add the **Message listener** source to consume a message endpoint: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Message listener** source to the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish.** 3. In the **Group ID** field, enter `${consumer.groupId}`. 4. In the **Topic Subscription Patterns** field, select **Edit inline** and then click the green plus sign. 5. Enter the value `${topic1}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector source.

Connector configuration

Global configuration you just created.

The following image shows an example of the **Message listener** configuration in the properties window: ![Message listener properties window configuration](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-message-listener-config.png) ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Message listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

"New message arrived: " payload ", key:" attributes.key ", partition:" attributes.partition ", offset:" ++ attributes.offset

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-logger-config-consume.png) ## XML for This Example Paste this code into the Studio XML editor to quickly load the flow for this example into your Mule app: ```xml ``` ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Publish and Then Consume a Topic with Kafka Connector > For the full documentation index, see: https://docs.mulesoft.com/llms.txt This example shows how to use two Apache Kafka Connector operations, **Publish** and **Consume**, to publish a message to Apache Kafka and then retrieve it. The example contains two flows: - The first flow is the Producer flow, which publishes a message to Apache Kafka. - The second flow is the Consumer flow, which consumes a message from Apache Kafka. ![Apache Kafka Connector Producer flow in Studio](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-producer-studio-flow.png) ![Apache Kafka Connector Consumer flow in Studio](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-consumer-studio-flow.png) ## Configure the First Flow Configure the first flow that publishes a message to Apache Kafka. Creating the first flow involves configuring a **Listener** component, a **Logger** component, a **Publish** operation, and a **Set Payload** transformer. ### Configure the HTTP Listener Configure the **Listener** component to initiate a Mule flow when a call is made to the `/pushMessage` path: 1. Create a new Mule project in Studio. 2. From the **Mule Palette** view, select **HTTP** and drag the **Listener** component to the canvas. 3. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 4. Accept the defaults. 5. In the properties window, set the **Path** field value to `/pushMessage`. ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

#["Message: '" payload.message "' is going to be published to topic: '" payload.topic "'."]

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-logger-config-publish.png) ### Add the Publish Operation Add the **Publish** operation to publish a message to the specified Kafka topic: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Publish** operation to the right of **Logger**. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector operation.

Connector configuration

Global configuration you just created.

Topic

\#[payload.topic]

Key

#[now()]

Message

payload

The following image shows an example of the **Publish** configuration in the properties window: ![Publish properties window configuration](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-publish.png) ### Add the Set Payload Transformer Add the **Set Payload** transformer to push the response builder: 1. From the **Mule Palette** view, select **Core** and drag the **Set Payload** transformer to the right of **Publish**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the transformer, such as Push response builder.

Value

Message successfully sent to Apache Kafka topic.

The following image shows an example of the **Set Payload** configuration in the properties window: ![Set Payload properties window configuration](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-set-payload.png) ## Configure the Second Flow Configure the second flow that consumes a message from Apache Kafka. Creating the second flow involves configuring a **Message listener** source and a **Logger** component. ### Add the Message Listener Source Add the **Message listener** source to consume a message endpoint: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Message listener** source to the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a global element. 3. Configure the global element as follows: 1. In the **Bootstrap Server URLs** field, select **Edit inline** and then click the green plus sign. 2. Enter the value `${config.basic.bootstrapServers}` and click **Finish.** 3. In the **Group ID** field, enter `${consumer.groupId}`. 4. In the **Topic Subscription Patterns** field, select **Edit inline** and then click the green plus sign. 5. Enter the value `${topic1}` and click **Finish**. 4. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the connector source.

Connector configuration

Global configuration you just created.

The following image shows an example of the **Message listener** configuration in the properties window: ![Message listener properties window configuration](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-message-listener-config.png) ### Add the Logger Component Add the **Logger** component to display the response in the Mule console: 1. From the **Mule Palette** view, select **Core** and drag a **Logger** component to the right of **Message listener**. 2. In the properties window, configure the following fields:
FieldValue

Display Name

Name for the logger.

Message

"New message arrived: " payload ", key:" attributes.key ", partition:" attributes.partition ", offset:" ++ attributes.offset

Level

INFO (Default)

The following image shows an example of the **Logger** configuration in the properties window: ![Logger properties window configuration](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-logger-config-consume.png) ## XML for This Example Paste this code into the Studio XML editor to quickly load the flow for this example into your Mule app: ```xml ``` ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.10 --- # Publish Multiple Messages with Bulk Publish: Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to use the **Bulk Publish** operation (``) of Kafka Connector to publish several messages to Apache Kafka with a single operation call. Imagine you receive a list of orders in your Mule app and want to publish each order as its own Kafka message, without calling **Publish** once per order. **Bulk Publish** takes an array of ready-made messages and sends all of them in one call. ## Understand the KafkaMessage Object **Publish** lets you set **Topic**, **Partition**, and **Key** directly as fields on the operation, because it only sends one message. **Bulk Publish** sends many messages at once, and each one can go to a different topic or partition, so instead of operation fields, you build an array where every entry carries its own routing information. Each entry of that array is a `KafkaMessage`, made of two parts:
FieldDescription

attributes

Where this message goes: topic (required), partition (required, a number: 0 if you don’t need to target a specific one), key (optional).

payload

The content of the message.

## Build the Array of KafkaMessage Objects with DataWeave `KafkaMessage` is a Java object, so a plain DataWeave object like `{topic: "orders"}` doesn’t work on its own. Tell DataWeave which Java class to build by writing the object as usual and tagging it with `as Object {class: "…​"}`. For example, if the incoming payload is an array of orders like this: ```json [ { "orderId": "1001", "description": "2 units of SKU-42" }, { "orderId": "1002", "description": "1 unit of SKU-77" } ] ``` The following DataWeave script turns that array into an array of `KafkaMessage` objects ready to publish: ```dataweave %dw 2.0 output application/java --- payload map (order) -> { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" } ``` > [!NOTE] Unlike **Publish**, **Bulk Publish** has no "let Kafka choose the partition" option. > `partition` always takes a real partition number, and `0` means partition 0, not "unspecified." To > spread messages across a target topic that has more than one partition, calculate the partition > per message yourself (for example, hashing the key against the topic’s partition count) instead of > sending every message to `0`. Set the **Messages** field of **Bulk Publish** to this expression (or reference the variable that holds its result, if you built it in an earlier **Transform Message** step). ## Configure the Bulk Publish Operation To configure the **Bulk Publish** operation in Anypoint Studio: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Bulk Publish** operation onto the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a **Producer** global element, or select an existing one. 3. Set the **Messages** field to the DataWeave expression that builds the array of `KafkaMessage` objects, for example `#[payload]`. **Bulk Publish** returns an array with one `KafkaMessageMetadata` entry per message that was published successfully, with `topic`, `partition`, and `offset` for each one. Because every message is sent independently, this array isn’t guaranteed to come back in the same order as the input array. Use the `topic`/`partition`/`offset` in each entry, not its position, to identify a message. ## XML for This Example Paste this code into the Studio XML editor to load the flow for this example into your Mule app. A **Scheduler** triggers the flow periodically. In a real app, replace the fixed list of orders in **Set orders payload** with wherever your orders actually come from (a database query, an HTTP call, and so on): ```xml { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" }]]> #[payload] ``` ## Handle Publishing Errors Because **Bulk Publish** sends every message independently, some messages in the array can succeed while others fail in the same call. For example, one message might point to a topic that doesn’t exist. Wrap the operation in a `try` to catch failures without losing the messages that went through: ```xml #[payload] ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-examples) - [Bulk Publish Reference](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-reference#bulkPublish) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.11 --- # Publish Multiple Messages with Bulk Publish: Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to use the **Bulk Publish** operation (``) of Kafka Connector to publish several messages to Apache Kafka with a single operation call. Imagine you receive a list of orders in your Mule app and want to publish each order as its own Kafka message, without calling **Publish** once per order. **Bulk Publish** takes an array of ready-made messages and sends all of them in one call. ## Understand the KafkaMessage Object **Publish** lets you set **Topic**, **Partition**, and **Key** directly as fields on the operation, because it only sends one message. **Bulk Publish** sends many messages at once, and each one can go to a different topic or partition, so instead of operation fields, you build an array where every entry carries its own routing information. Each entry of that array is a `KafkaMessage`, made of two parts:
FieldDescription

attributes

Where this message goes: topic (required), partition (required, a number: 0 if you don’t need to target a specific one), key (optional).

payload

The content of the message.

## Build the Array of KafkaMessage Objects with DataWeave `KafkaMessage` is a Java object, so a plain DataWeave object like `{topic: "orders"}` doesn’t work on its own. Tell DataWeave which Java class to build by writing the object as usual and tagging it with `as Object {class: "…​"}`. For example, if the incoming payload is an array of orders like this: ```json [ { "orderId": "1001", "description": "2 units of SKU-42" }, { "orderId": "1002", "description": "1 unit of SKU-77" } ] ``` The following DataWeave script turns that array into an array of `KafkaMessage` objects ready to publish: ```dataweave %dw 2.0 output application/java --- payload map (order) -> { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" } ``` > [!NOTE] Unlike **Publish**, **Bulk Publish** has no "let Kafka choose the partition" option. > `partition` always takes a real partition number, and `0` means partition 0, not "unspecified." To > spread messages across a target topic that has more than one partition, calculate the partition > per message yourself (for example, hashing the key against the topic’s partition count) instead of > sending every message to `0`. Set the **Messages** field of **Bulk Publish** to this expression (or reference the variable that holds its result, if you built it in an earlier **Transform Message** step). ## Configure the Bulk Publish Operation To configure the **Bulk Publish** operation in Anypoint Studio: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Bulk Publish** operation onto the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a **Producer** global element, or select an existing one. 3. Set the **Messages** field to the DataWeave expression that builds the array of `KafkaMessage` objects, for example `#[payload]`. **Bulk Publish** returns an array with one `KafkaMessageMetadata` entry per message that was published successfully, with `topic`, `partition`, and `offset` for each one. Because every message is sent independently, this array isn’t guaranteed to come back in the same order as the input array. Use the `topic`/`partition`/`offset` in each entry, not its position, to identify a message. ## XML for This Example Paste this code into the Studio XML editor to load the flow for this example into your Mule app. A **Scheduler** triggers the flow periodically. In a real app, replace the fixed list of orders in **Set orders payload** with wherever your orders actually come from (a database query, an HTTP call, and so on): ```xml { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" }]]> #[payload] ``` ## Handle Publishing Errors Because **Bulk Publish** sends every message independently, some messages in the array can succeed while others fail in the same call. For example, one message might point to a topic that doesn’t exist. Wrap the operation in a `try` to catch failures without losing the messages that went through: ```xml #[payload] ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-examples) - [Bulk Publish Reference](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-reference#bulkPublish) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.12 --- # Publish Multiple Messages with Bulk Publish: Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to use the **Bulk Publish** operation (``) of Kafka Connector to publish several messages to Apache Kafka with a single operation call. Imagine you receive a list of orders in your Mule app and want to publish each order as its own Kafka message, without calling **Publish** once per order. **Bulk Publish** takes an array of ready-made messages and sends all of them in one call. ## Understand the KafkaMessage Object **Publish** lets you set **Topic**, **Partition**, and **Key** directly as fields on the operation, because it only sends one message. **Bulk Publish** sends many messages at once, and each one can go to a different topic or partition, so instead of operation fields, you build an array where every entry carries its own routing information. Each entry of that array is a `KafkaMessage`, made of two parts:
FieldDescription

attributes

Where this message goes: topic (required), partition (required, a number: 0 if you don’t need to target a specific one), key (optional).

payload

The content of the message.

## Build the Array of KafkaMessage Objects with DataWeave `KafkaMessage` is a Java object, so a plain DataWeave object like `{topic: "orders"}` doesn’t work on its own. Tell DataWeave which Java class to build by writing the object as usual and tagging it with `as Object {class: "…​"}`. For example, if the incoming payload is an array of orders like this: ```json [ { "orderId": "1001", "description": "2 units of SKU-42" }, { "orderId": "1002", "description": "1 unit of SKU-77" } ] ``` The following DataWeave script turns that array into an array of `KafkaMessage` objects ready to publish: ```dataweave %dw 2.0 output application/java --- payload map (order) -> { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" } ``` > [!NOTE] Unlike **Publish**, **Bulk Publish** has no "let Kafka choose the partition" option. > `partition` always takes a real partition number, and `0` means partition 0, not "unspecified." To > spread messages across a target topic that has more than one partition, calculate the partition > per message yourself (for example, hashing the key against the topic’s partition count) instead of > sending every message to `0`. Set the **Messages** field of **Bulk Publish** to this expression (or reference the variable that holds its result, if you built it in an earlier **Transform Message** step). ## Configure the Bulk Publish Operation To configure the **Bulk Publish** operation in Anypoint Studio: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Bulk Publish** operation onto the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a **Producer** global element, or select an existing one. 3. Set the **Messages** field to the DataWeave expression that builds the array of `KafkaMessage` objects, for example `#[payload]`. **Bulk Publish** returns an array with one `KafkaMessageMetadata` entry per message that was published successfully, with `topic`, `partition`, and `offset` for each one. Because every message is sent independently, this array isn’t guaranteed to come back in the same order as the input array. Use the `topic`/`partition`/`offset` in each entry, not its position, to identify a message. ## XML for This Example Paste this code into the Studio XML editor to load the flow for this example into your Mule app. A **Scheduler** triggers the flow periodically. In a real app, replace the fixed list of orders in **Set orders payload** with wherever your orders actually come from (a database query, an HTTP call, and so on): ```xml { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" }]]> #[payload] ``` ## Handle Publishing Errors Because **Bulk Publish** sends every message independently, some messages in the array can succeed while others fail in the same call. For example, one message might point to a topic that doesn’t exist. Wrap the operation in a `try` to catch failures without losing the messages that went through: ```xml #[payload] ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-examples) - [Bulk Publish Reference](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-reference#bulkPublish) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.13 --- # Publish Multiple Messages with Bulk Publish: Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to use the **Bulk Publish** operation (``) of Kafka Connector to publish several messages to Apache Kafka with a single operation call. Imagine you receive a list of orders in your Mule app and want to publish each order as its own Kafka message, without calling **Publish** once per order. **Bulk Publish** takes an array of ready-made messages and sends all of them in one call. ## Understand the KafkaMessage Object **Publish** lets you set **Topic**, **Partition**, and **Key** directly as fields on the operation, because it only sends one message. **Bulk Publish** sends many messages at once, and each one can go to a different topic or partition, so instead of operation fields, you build an array where every entry carries its own routing information. Each entry of that array is a `KafkaMessage`, made of two parts:
FieldDescription

attributes

Where this message goes: topic (required), partition (required, a number: 0 if you don’t need to target a specific one), key (optional).

payload

The content of the message.

## Build the Array of KafkaMessage Objects with DataWeave `KafkaMessage` is a Java object, so a plain DataWeave object like `{topic: "orders"}` doesn’t work on its own. Tell DataWeave which Java class to build by writing the object as usual and tagging it with `as Object {class: "…​"}`. For example, if the incoming payload is an array of orders like this: ```json [ { "orderId": "1001", "description": "2 units of SKU-42" }, { "orderId": "1002", "description": "1 unit of SKU-77" } ] ``` The following DataWeave script turns that array into an array of `KafkaMessage` objects ready to publish: ```dataweave %dw 2.0 output application/java --- payload map (order) -> { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" } ``` > [!NOTE] Unlike **Publish**, **Bulk Publish** has no "let Kafka choose the partition" option. > `partition` always takes a real partition number, and `0` means partition 0, not "unspecified." To > spread messages across a target topic that has more than one partition, calculate the partition > per message yourself (for example, hashing the key against the topic’s partition count) instead of > sending every message to `0`. Set the **Messages** field of **Bulk Publish** to this expression (or reference the variable that holds its result, if you built it in an earlier **Transform Message** step). ## Configure the Bulk Publish Operation To configure the **Bulk Publish** operation in Anypoint Studio: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Bulk Publish** operation onto the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a **Producer** global element, or select an existing one. 3. Set the **Messages** field to the DataWeave expression that builds the array of `KafkaMessage` objects, for example `#[payload]`. **Bulk Publish** returns an array with one `KafkaMessageMetadata` entry per message that was published successfully, with `topic`, `partition`, and `offset` for each one. Because every message is sent independently, this array isn’t guaranteed to come back in the same order as the input array. Use the `topic`/`partition`/`offset` in each entry, not its position, to identify a message. ## XML for This Example Paste this code into the Studio XML editor to load the flow for this example into your Mule app. A **Scheduler** triggers the flow periodically. In a real app, replace the fixed list of orders in **Set orders payload** with wherever your orders actually come from (a database query, an HTTP call, and so on): ```xml { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" }]]> #[payload] ``` ## Handle Publishing Errors Because **Bulk Publish** sends every message independently, some messages in the array can succeed while others fail in the same call. For example, one message might point to a topic that doesn’t exist. Wrap the operation in a `try` to catch failures without losing the messages that went through: ```xml #[payload] ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-examples) - [Bulk Publish Reference](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-reference#bulkPublish) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.14 --- # Publish Multiple Messages with Bulk Publish: Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to use the **Bulk Publish** operation (``) of Kafka Connector to publish several messages to Apache Kafka with a single operation call. Imagine you receive a list of orders in your Mule app and want to publish each order as its own Kafka message, without calling **Publish** once per order. **Bulk Publish** takes an array of ready-made messages and sends all of them in one call. ## Understand the KafkaMessage Object **Publish** lets you set **Topic**, **Partition**, and **Key** directly as fields on the operation, because it only sends one message. **Bulk Publish** sends many messages at once, and each one can go to a different topic or partition, so instead of operation fields, you build an array where every entry carries its own routing information. Each entry of that array is a `KafkaMessage`, made of two parts:
FieldDescription

attributes

Where this message goes: topic (required), partition (required, a number: 0 if you don’t need to target a specific one), key (optional).

payload

The content of the message.

## Build the Array of KafkaMessage Objects with DataWeave `KafkaMessage` is a Java object, so a plain DataWeave object like `{topic: "orders"}` doesn’t work on its own. Tell DataWeave which Java class to build by writing the object as usual and tagging it with `as Object {class: "…​"}`. For example, if the incoming payload is an array of orders like this: ```json [ { "orderId": "1001", "description": "2 units of SKU-42" }, { "orderId": "1002", "description": "1 unit of SKU-77" } ] ``` The following DataWeave script turns that array into an array of `KafkaMessage` objects ready to publish: ```dataweave %dw 2.0 output application/java --- payload map (order) -> { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" } ``` > [!NOTE] Unlike **Publish**, **Bulk Publish** has no "let Kafka choose the partition" option. > `partition` always takes a real partition number, and `0` means partition 0, not "unspecified." To > spread messages across a target topic that has more than one partition, calculate the partition > per message yourself (for example, hashing the key against the topic’s partition count) instead of > sending every message to `0`. Set the **Messages** field of **Bulk Publish** to this expression (or reference the variable that holds its result, if you built it in an earlier **Transform Message** step). ## Configure the Bulk Publish Operation To configure the **Bulk Publish** operation in Anypoint Studio: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Bulk Publish** operation onto the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a **Producer** global element, or select an existing one. 3. Set the **Messages** field to the DataWeave expression that builds the array of `KafkaMessage` objects, for example `#[payload]`. **Bulk Publish** returns an array with one `KafkaMessageMetadata` entry per message that was published successfully, with `topic`, `partition`, and `offset` for each one. Because every message is sent independently, this array isn’t guaranteed to come back in the same order as the input array. Use the `topic`/`partition`/`offset` in each entry, not its position, to identify a message. ## XML for This Example Paste this code into the Studio XML editor to load the flow for this example into your Mule app. A **Scheduler** triggers the flow periodically. In a real app, replace the fixed list of orders in **Set orders payload** with wherever your orders actually come from (a database query, an HTTP call, and so on): ```xml { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" }]]> #[payload] ``` ## Handle Publishing Errors Because **Bulk Publish** sends every message independently, some messages in the array can succeed while others fail in the same call. For example, one message might point to a topic that doesn’t exist. Wrap the operation in a `try` to catch failures without losing the messages that went through: ```xml #[payload] ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-examples) - [Bulk Publish Reference](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-reference#bulkPublish) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Publish Multiple Messages with Bulk Publish: Example > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to use the **Bulk Publish** operation (``) of Kafka Connector to publish several messages to Apache Kafka with a single operation call. Imagine you receive a list of orders in your Mule app and want to publish each order as its own Kafka message, without calling **Publish** once per order. **Bulk Publish** takes an array of ready-made messages and sends all of them in one call. ## Understand the KafkaMessage Object **Publish** lets you set **Topic**, **Partition**, and **Key** directly as fields on the operation, because it only sends one message. **Bulk Publish** sends many messages at once, and each one can go to a different topic or partition, so instead of operation fields, you build an array where every entry carries its own routing information. Each entry of that array is a `KafkaMessage`, made of two parts:
FieldDescription

attributes

Where this message goes: topic (required), partition (required, a number: 0 if you don’t need to target a specific one), key (optional).

payload

The content of the message.

## Build the Array of KafkaMessage Objects with DataWeave `KafkaMessage` is a Java object, so a plain DataWeave object like `{topic: "orders"}` doesn’t work on its own. Tell DataWeave which Java class to build by writing the object as usual and tagging it with `as Object {class: "…​"}`. For example, if the incoming payload is an array of orders like this: ```json [ { "orderId": "1001", "description": "2 units of SKU-42" }, { "orderId": "1002", "description": "1 unit of SKU-77" } ] ``` The following DataWeave script turns that array into an array of `KafkaMessage` objects ready to publish: ```dataweave %dw 2.0 output application/java --- payload map (order) -> { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" } ``` > [!NOTE] Unlike **Publish**, **Bulk Publish** has no "let Kafka choose the partition" option. > `partition` always takes a real partition number, and `0` means partition 0, not "unspecified." To > spread messages across a target topic that has more than one partition, calculate the partition > per message yourself (for example, hashing the key against the topic’s partition count) instead of > sending every message to `0`. Set the **Messages** field of **Bulk Publish** to this expression (or reference the variable that holds its result, if you built it in an earlier **Transform Message** step). ## Configure the Bulk Publish Operation To configure the **Bulk Publish** operation in Anypoint Studio: 1. From the **Mule Palette** view, select **Apache Kafka** and drag the **Bulk Publish** operation onto the canvas. 2. In the properties window, click **+** next to the **Connector configuration** field to add a **Producer** global element, or select an existing one. 3. Set the **Messages** field to the DataWeave expression that builds the array of `KafkaMessage` objects, for example `#[payload]`. **Bulk Publish** returns an array with one `KafkaMessageMetadata` entry per message that was published successfully, with `topic`, `partition`, and `offset` for each one. Because every message is sent independently, this array isn’t guaranteed to come back in the same order as the input array. Use the `topic`/`partition`/`offset` in each entry, not its position, to identify a message. ## XML for This Example Paste this code into the Studio XML editor to load the flow for this example into your Mule app. A **Scheduler** triggers the flow periodically. In a real app, replace the fixed list of orders in **Set orders payload** with wherever your orders actually come from (a database query, an HTTP call, and so on): ```xml { attributes: { topic: "orders", partition: 0, key: order.orderId }, payload: order.description } as Object { class: "com.mulesoft.connectors.kafka.api.operation.KafkaMessage" }]]> #[payload] ``` ## Handle Publishing Errors Because **Bulk Publish** sends every message independently, some messages in the array can succeed while others fail in the same call. For example, one message might point to a topic that doesn’t exist. Wrap the operation in a `try` to catch failures without losing the messages that went through: ```xml #[payload] ``` ## See Also - [Kafka Connector Examples](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-examples) - [Bulk Publish Reference](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-reference#bulkPublish) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.10 --- # Troubleshooting Apache Kafka Connector 4.10 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt To troubleshoot Anypoint Connector for Apache Kafka (Apache Kafka Connector), become familiar with the information about interpreting commonly thrown exception messages. ## Enable SSL Logging Enable SSL logs to investigate an issue related to TLS communications. Set your SSL log level depending on your required level of detail: 1. Enable only SSL log details: `javax.net.debug=ssl` 2. Enable only handshaking details. (The handshake protocol is a series of messages exchanged over the record protocol.) `javax.net.debug=ssl:handshake` 3. Enable handshaking details and SSL details at the same time: `javax.net.debug=ssl, handshake` 4. Enable the dumping of all details and traffic data: `javax.net.debug=all` > [!NOTE] This option is very verbose and under normal circumstances is not necessary. > [!WARNING] SSL logging results in a performance impact for HTTPS or other TLS connections. Enable > SSL logging only to troubleshoot a specific issue and do not enable it for extended periods. SSL > logging produces a significant amount of log messages and can inundate your log file if it is left > enabled and unattended. To enable SSL logs: Set the debug parameter as an argument in the runtime configuration for the application. 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and add `-M-Djavax.net.debug=ssl` in the **VM arguments** section. ## Enable Verbose Exception Logging Enable verbose exception logs to show a complete stack trace of the error instead of the default truncated output: 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and append the arguments in the **VM arguments** section to add the `mule.verbose.exceptions` property. For example: `-XX:PermSize=128M -XX:MaxPermSize=256M -Dmule.verbose.exceptions=true` ## Enable Verbose Logging To get a better understanding of why an application’s interaction with Kafka Connector is failing, temporarily enable verbose logging for the connector. Remember to always disable enhanced verbosity after troubleshooting, because it can affect your Mule application’s performance. To enable verbose logging in the configuration file: 1. Access Anypoint Studio and navigate to the **Package Explorer** view. 2. Open your application’s project name. 3. Open the `src/main/resources` folder. 4. Open the `log4j2.xml` file inside the folder. 5. Add an `` tag inside the `` tag: ```xml ... ... ``` 6. Save your application changes. 7. Click the project name in **Package Explorer** and then click **Run** > **Run As** > **Mule Application**. ## Troubleshoot Common Errors Here is a list of the common Apache Kafka Connector errors and how to resolve them: - [Failed to Load SSL Keystore](#failed-to-load-ssl-keystore) ### Failed to Load SSL Keystore When the Kafka Connector configurations fail to load the required truststore files from the classpath, you receive the following error: ```text org.mule.runtime.api.connection.ConnectionException: Failed to load SSL keystore ``` The Kafka Connector configurations bundled with a dependency fail to establish connections even when the required truststore files are available in the classpath. To resolve this error: Add the required truststore files to the main application that uses the Kafka Connector configurations. ## Troubleshooting clientDNSLookUp Property Since Kafka Connector version 3.0.0, the default value for the **DNS Lookup** field was removed to keep the connector backwards compatible. Kafka Connector keeps the same values as the previous version, but now each value corresponds to the following properties:
ValueProperty

DEFAULT

use_all_dns_ips

USE_ALL_DNS_IPS

use_all_dns_ips

RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

resolve_canonical_bootstrap_servers_only

## Troubleshooting Publishing Null Values When the **Publish** operation receives null as input, the operation publishes an empty byte array. To avoid this issue, the connector enables you to set the `mule.kafka.publish.useNull` boolean system property. - When set to `true`, the connector publishes a message with null as a value. - When set to `false` or not set at all, the connector publishes a message with an empty byte array value. For further information on how to configure properties, refer to the [Use system properties](https://docs.mulesoft.com/mule-runtime/latest/mule-app-properties-system) ## Understand Commonly Thrown Exceptions Here is a list of commonly thrown exception messages and how to interpret them: - KAFKA:ALREADY\_COMMITTED A commit fails because the commit is already performed. - KAFKA:AUTHENTICATION\_ERROR(SECURITY) An authentication error is thrown. - KAFKA:AUTHORIZATION\_ERROR(CLIENT\_SECURITY) An authorization error occurred due to the client configuration or permissions. - KAFKA:COMMIT\_FAILED A commit fails with an unrecoverable error. - KAFKA:CONNECTIVITY Either a connection could not be established or there is a connectivity error. - KAFKA:ILLEGAL\_STATE The consumer is not subscribed to any topics, and no manually assigned partitions are set to consume from. - KAFKA:INPUT\_TOO\_LARGE The input is too large for the broker. - KAFKA:INVALID\_ACK\_MODE An acknowledgement (ACK) or negative acknowledgement (NACK) operation is run over a consumer that is not in \`MANUAL\` mode. - KAFKA:INVALID\_CONFIGURATION An invalid configuration value is set for the consumer or the producer. - KAFKA:INVALID\_CONNECTION The referenced connection is invalid. - KAFKA:INVALID\_INPUT The parameters set for the operation are incorrect. - KAFKA:INVALID\_OFFSET The partition number set for the operation is less than \`0\`, is above the number of partitions, or is an invalid value. - KAFKA:INVALID\_TOPIC The topic selected for the operation is invalid, and there is no automatic generation of said topic. This might also imply an invalid character in the topic name. - KAFKA:INVALID\_TOPIC\_PARTITION The topic partition combination selected for the operation is invalid, and there is no automatic generation of topics. This might also imply an invalid character in the topic name, or a non-existing partition. - KAFKA:NO\_POLL\_MADE No poll was made. - KAFKA:NOT\_FOUND For consume operations, the poll does not bring any records for the configured timeouts. - KAFKA:OUT\_OF\_RANGE There is no reset policy defined, and the offsets for these partitions are either larger or smaller than the range of offsets the server has for the given partition. - KAFKA:PREVIOUS\_ASSIGNATION subscribe() is called previously with topics or pattern without a subsequent call to unsubscribe(). - KAFKA:PRODUCER\_FENCED Another producer with the same \`transactional.id\` is active. - KAFKA:RETRY\_EXHAUSTED The maximum number of retries for the operation is reached. - KAFKA:SESSION\_NOT\_FOUND No associated session is found when executing an operation. - KAFKA:TIMEOUT A specific request takes longer than the configuration timeout values. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Apache Kafka Connector Reference](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-reference) - [Salesforce Help](https://help.salesforce.com) - [Salesforce Help](https://help.salesforce.com) - [Known Issues](https://help.salesforce.com/s/issues) --- product: Apache Kafka Connector version: 4.11 --- # Troubleshooting Apache Kafka Connector 4.11 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt To troubleshoot Anypoint Connector for Apache Kafka (Apache Kafka Connector), become familiar with the information about interpreting commonly thrown exception messages. ## Enable SSL Logging Enable SSL logs to investigate an issue related to TLS communications. Set your SSL log level depending on your required level of detail: 1. Enable only SSL log details: `javax.net.debug=ssl` 2. Enable only handshaking details. (The handshake protocol is a series of messages exchanged over the record protocol.) `javax.net.debug=ssl:handshake` 3. Enable handshaking details and SSL details at the same time: `javax.net.debug=ssl, handshake` 4. Enable the dumping of all details and traffic data: `javax.net.debug=all` > [!NOTE] This option is very verbose and under normal circumstances is not necessary. > [!WARNING] SSL logging results in a performance impact for HTTPS or other TLS connections. Enable > SSL logging only to troubleshoot a specific issue and do not enable it for extended periods. SSL > logging produces a significant amount of log messages and can inundate your log file if it is left > enabled and unattended. To enable SSL logs: Set the debug parameter as an argument in the runtime configuration for the application. 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and add `-M-Djavax.net.debug=ssl` in the **VM arguments** section. ## Enable Verbose Exception Logging Enable verbose exception logs to show a complete stack trace of the error instead of the default truncated output: 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and append the arguments in the **VM arguments** section to add the `mule.verbose.exceptions` property. For example: `-XX:PermSize=128M -XX:MaxPermSize=256M -Dmule.verbose.exceptions=true` ## Enable Verbose Logging To get a better understanding of why an application’s interaction with Kafka Connector is failing, temporarily enable verbose logging for the connector. Remember to always disable enhanced verbosity after troubleshooting, because it can affect your Mule application’s performance. To enable verbose logging in the configuration file: 1. Access Anypoint Studio and navigate to the **Package Explorer** view. 2. Open your application’s project name. 3. Open the `src/main/resources` folder. 4. Open the `log4j2.xml` file inside the folder. 5. Add an `` tag inside the `` tag: ```xml ... ... ``` 6. Save your application changes. 7. Click the project name in **Package Explorer** and then click **Run** > **Run As** > **Mule Application**. ## Enable FIPS Mode Enable FIPS mode by setting the `com.mulesoft.connectors.allow-registration-of-sasl-security-provider-in-fips=true` flag to true. This adds the `com.sun.security.sasl.Provider` security provider in the FIPS environment. ## Failed to Load SSL Keystore When the Kafka Connector configurations fail to load the required truststore files from the classpath, you receive the following error: ```text org.mule.runtime.api.connection.ConnectionException: Failed to load SSL keystore ``` The Kafka Connector configurations bundled with a dependency fail to establish connections even when the required truststore files are available in the classpath. To resolve this error: Add the required truststore files to the main application that uses the Kafka Connector configurations. ## Troubleshooting clientDNSLookUp Property Since Kafka Connector version 3.0.0, the default value for the **DNS Lookup** field was removed to keep the connector backwards compatible. Kafka Connector keeps the same values as the previous version, but now each value corresponds to the following properties:
ValueProperty

DEFAULT

use_all_dns_ips

USE_ALL_DNS_IPS

use_all_dns_ips

RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

resolve_canonical_bootstrap_servers_only

## Troubleshooting Publishing Null Values When the **Publish** operation receives null as input, the operation publishes an empty byte array. To avoid this issue, the connector enables you to set the `mule.kafka.publish.useNull` boolean system property. - When set to `true`, the connector publishes a message with null as a value. - When set to `false` or not set at all, the connector publishes a message with an empty byte array value. For further information on how to configure properties, refer to the [Use system properties](https://docs.mulesoft.com/mule-runtime/latest/mule-app-properties-system) ## Understand Commonly Thrown Exceptions Here is a list of commonly thrown exception messages and how to interpret them: - KAFKA:ALREADY\_COMMITTED A commit fails because the commit is already performed. - KAFKA:AUTHENTICATION\_ERROR(SECURITY) An authentication error is thrown. - KAFKA:AUTHORIZATION\_ERROR(CLIENT\_SECURITY) An authorization error occurred due to the client configuration or permissions. - KAFKA:COMMIT\_FAILED A commit fails with an unrecoverable error. - KAFKA:CONNECTIVITY Either a connection could not be established or there is a connectivity error. - KAFKA:ILLEGAL\_STATE The consumer is not subscribed to any topics, and no manually assigned partitions are set to consume from. - KAFKA:INPUT\_TOO\_LARGE The input is too large for the broker. - KAFKA:INVALID\_ACK\_MODE An acknowledgement (ACK) or negative acknowledgement (NACK) operation is run over a consumer that is not in \`MANUAL\` mode. - KAFKA:INVALID\_CONFIGURATION An invalid configuration value is set for the consumer or the producer. - KAFKA:INVALID\_CONNECTION The referenced connection is invalid. - KAFKA:INVALID\_INPUT The parameters set for the operation are incorrect. - KAFKA:INVALID\_OFFSET The partition number set for the operation is less than \`0\`, is above the number of partitions, or is an invalid value. - KAFKA:INVALID\_TOPIC The topic selected for the operation is invalid, and there is no automatic generation of said topic. This might also imply an invalid character in the topic name. - KAFKA:INVALID\_TOPIC\_PARTITION The topic partition combination selected for the operation is invalid, and there is no automatic generation of topics. This might also imply an invalid character in the topic name, or a non-existing partition. - KAFKA:NO\_POLL\_MADE No poll was made. - KAFKA:NOT\_FOUND For consume operations, the poll does not bring any records for the configured timeouts. - KAFKA:OUT\_OF\_RANGE There is no reset policy defined, and the offsets for these partitions are either larger or smaller than the range of offsets the server has for the given partition. - KAFKA:PREVIOUS\_ASSIGNATION subscribe() is called previously with topics or pattern without a subsequent call to unsubscribe(). - KAFKA:PRODUCER\_FENCED Another producer with the same \`transactional.id\` is active. - KAFKA:RETRY\_EXHAUSTED The maximum number of retries for the operation is reached. - KAFKA:SESSION\_NOT\_FOUND No associated session is found when executing an operation. - KAFKA:TIMEOUT A specific request takes longer than the configuration timeout values. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Apache Kafka Connector Reference](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-reference) - [Salesforce Help](https://help.salesforce.com) - [Salesforce Help](https://help.salesforce.com) - [Known Issues](https://help.salesforce.com/s/issues) --- product: Apache Kafka Connector version: 4.12 --- # Troubleshooting Apache Kafka Connector 4.12 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt To troubleshoot Anypoint Connector for Apache Kafka (Apache Kafka Connector), become familiar with the information about interpreting commonly thrown exception messages. ## Enable SSL Logging Enable SSL logs to investigate an issue related to TLS communications. Set your SSL log level depending on your required level of detail: 1. Enable only SSL log details: `javax.net.debug=ssl` 2. Enable only handshaking details. (The handshake protocol is a series of messages exchanged over the record protocol.) `javax.net.debug=ssl:handshake` 3. Enable handshaking details and SSL details at the same time: `javax.net.debug=ssl, handshake` 4. Enable the dumping of all details and traffic data: `javax.net.debug=all` > [!NOTE] This option is very verbose and under normal circumstances is not necessary. > [!WARNING] SSL logging results in a performance impact for HTTPS or other TLS connections. Enable > SSL logging only to troubleshoot a specific issue and do not enable it for extended periods. SSL > logging produces a significant amount of log messages and can inundate your log file if it is left > enabled and unattended. To enable SSL logs: Set the debug parameter as an argument in the runtime configuration for the application. 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and add `-M-Djavax.net.debug=ssl` in the **VM arguments** section. ## Enable Verbose Exception Logging Enable verbose exception logs to show a complete stack trace of the error instead of the default truncated output: 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and append the arguments in the **VM arguments** section to add the `mule.verbose.exceptions` property. For example: `-XX:PermSize=128M -XX:MaxPermSize=256M -Dmule.verbose.exceptions=true` ## Enable Verbose Logging To get a better understanding of why an application’s interaction with Kafka Connector is failing, temporarily enable verbose logging for the connector. Remember to always disable enhanced verbosity after troubleshooting, because it can affect your Mule application’s performance. To enable verbose logging in the configuration file: 1. Access Anypoint Studio and navigate to the **Package Explorer** view. 2. Open your application’s project name. 3. Open the `src/main/resources` folder. 4. Open the `log4j2.xml` file inside the folder. 5. Add an `` tag inside the `` tag: ```xml ... ... ``` 6. Save your application changes. 7. Click the project name in **Package Explorer** and then click **Run** > **Run As** > **Mule Application**. ## Enable FIPS Mode Enable FIPS mode by setting the `com.mulesoft.connectors.allow-registration-of-sasl-security-provider-in-fips=true` flag to true. This adds the `com.sun.security.sasl.Provider` security provider in the FIPS environment. ## Failed to Load SSL Keystore When the Kafka Connector configurations fail to load the required truststore files from the classpath, you receive the following error: ```text org.mule.runtime.api.connection.ConnectionException: Failed to load SSL keystore ``` The Kafka Connector configurations bundled with a dependency fail to establish connections even when the required truststore files are available in the classpath. To resolve this error: Add the required truststore files to the main application that uses the Kafka Connector configurations. ## Troubleshooting clientDNSLookUp Property Since Kafka Connector version 3.0.0, the default value for the **DNS Lookup** field was removed to keep the connector backwards compatible. Kafka Connector keeps the same values as the previous version, but now each value corresponds to the following properties:
ValueProperty

DEFAULT

use_all_dns_ips

USE_ALL_DNS_IPS

use_all_dns_ips

RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

resolve_canonical_bootstrap_servers_only

## Troubleshooting Publishing Null Values When the **Publish** operation receives null as input, the operation publishes an empty byte array. To avoid this issue, the connector enables you to set the `mule.kafka.publish.useNull` boolean system property. - When set to `true`, the connector publishes a message with null as a value. - When set to `false` or not set at all, the connector publishes a message with an empty byte array value. For further information on how to configure properties, refer to the [Use system properties](https://docs.mulesoft.com/mule-runtime/latest/mule-app-properties-system) ## Understand Commonly Thrown Exceptions Here is a list of commonly thrown exception messages and how to interpret them: - KAFKA:ALREADY\_COMMITTED A commit fails because the commit is already performed. - KAFKA:AUTHENTICATION\_ERROR(SECURITY) An authentication error is thrown. - KAFKA:AUTHORIZATION\_ERROR(CLIENT\_SECURITY) An authorization error occurred due to the client configuration or permissions. - KAFKA:COMMIT\_FAILED A commit fails with an unrecoverable error. - KAFKA:CONNECTIVITY Either a connection could not be established or there is a connectivity error. - KAFKA:ILLEGAL\_STATE The consumer is not subscribed to any topics, and no manually assigned partitions are set to consume from. - KAFKA:INPUT\_TOO\_LARGE The input is too large for the broker. - KAFKA:INVALID\_ACK\_MODE An acknowledgement (ACK) or negative acknowledgement (NACK) operation is run over a consumer that is not in \`MANUAL\` mode. - KAFKA:INVALID\_CONFIGURATION An invalid configuration value is set for the consumer or the producer. - KAFKA:INVALID\_CONNECTION The referenced connection is invalid. - KAFKA:INVALID\_INPUT The parameters set for the operation are incorrect. - KAFKA:INVALID\_OFFSET The partition number set for the operation is less than \`0\`, is above the number of partitions, or is an invalid value. - KAFKA:INVALID\_TOPIC The topic selected for the operation is invalid, and there is no automatic generation of said topic. This might also imply an invalid character in the topic name. - KAFKA:INVALID\_TOPIC\_PARTITION The topic partition combination selected for the operation is invalid, and there is no automatic generation of topics. This might also imply an invalid character in the topic name, or a non-existing partition. - KAFKA:NO\_POLL\_MADE No poll was made. - KAFKA:NOT\_FOUND For consume operations, the poll does not bring any records for the configured timeouts. - KAFKA:OUT\_OF\_RANGE There is no reset policy defined, and the offsets for these partitions are either larger or smaller than the range of offsets the server has for the given partition. - KAFKA:PREVIOUS\_ASSIGNATION subscribe() is called previously with topics or pattern without a subsequent call to unsubscribe(). - KAFKA:PRODUCER\_FENCED Another producer with the same \`transactional.id\` is active. - KAFKA:RETRY\_EXHAUSTED The maximum number of retries for the operation is reached. - KAFKA:SESSION\_NOT\_FOUND No associated session is found when executing an operation. - KAFKA:TIMEOUT A specific request takes longer than the configuration timeout values. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Apache Kafka Connector Reference](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-reference) - [Salesforce Help](https://help.salesforce.com) - [Salesforce Help](https://help.salesforce.com) - [Known Issues](https://help.salesforce.com/s/issues) --- product: Apache Kafka Connector version: 4.13 --- # Troubleshooting Apache Kafka Connector 4.13 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt To troubleshoot Anypoint Connector for Apache Kafka (Apache Kafka Connector), become familiar with the information about interpreting commonly thrown exception messages. ## Enable SSL Logging Enable SSL logs to investigate an issue related to TLS communications. Set your SSL log level depending on your required level of detail: 1. Enable only SSL log details: `javax.net.debug=ssl` 2. Enable only handshaking details. (The handshake protocol is a series of messages exchanged over the record protocol.) `javax.net.debug=ssl:handshake` 3. Enable handshaking details and SSL details at the same time: `javax.net.debug=ssl, handshake` 4. Enable the dumping of all details and traffic data: `javax.net.debug=all` > [!NOTE] This option is very verbose and under normal circumstances is not necessary. > [!WARNING] SSL logging results in a performance impact for HTTPS or other TLS connections. Enable > SSL logging only to troubleshoot a specific issue and do not enable it for extended periods. SSL > logging produces a significant amount of log messages and can inundate your log file if it is left > enabled and unattended. To enable SSL logs: Set the debug parameter as an argument in the runtime configuration for the application. 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and add `-M-Djavax.net.debug=ssl` in the **VM arguments** section. ## Enable Verbose Exception Logging Enable verbose exception logs to show a complete stack trace of the error instead of the default truncated output: 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and append the arguments in the **VM arguments** section to add the `mule.verbose.exceptions` property. For example: `-XX:PermSize=128M -XX:MaxPermSize=256M -Dmule.verbose.exceptions=true` ## Enable Verbose Logging To get a better understanding of why an application’s interaction with Kafka Connector is failing, temporarily enable verbose logging for the connector. Remember to always disable enhanced verbosity after troubleshooting, because it can affect your Mule application’s performance. To enable verbose logging in the configuration file: 1. Access Anypoint Studio and navigate to the **Package Explorer** view. 2. Open your application’s project name. 3. Open the `src/main/resources` folder. 4. Open the `log4j2.xml` file inside the folder. 5. Add an `` tag inside the `` tag: ```xml ... ... ``` 6. Save your application changes. 7. Click the project name in **Package Explorer** and then click **Run** > **Run As** > **Mule Application**. ## Enable FIPS Mode Enable FIPS mode by setting the `com.mulesoft.connectors.allow-registration-of-sasl-security-provider-in-fips=true` flag to true. This adds the `com.sun.security.sasl.Provider` security provider in the FIPS environment. ## Failed to Load SSL Keystore When the Kafka Connector configurations fail to load the required truststore files from the classpath, you receive the following error: ```text org.mule.runtime.api.connection.ConnectionException: Failed to load SSL keystore ``` The Kafka Connector configurations bundled with a dependency fail to establish connections even when the required truststore files are available in the classpath. To resolve this error: Add the required truststore files to the main application that uses the Kafka Connector configurations. ## Troubleshooting clientDNSLookUp Property Since Kafka Connector version 3.0.0, the default value for the **DNS Lookup** field was removed to keep the connector backwards compatible. Kafka Connector keeps the same values as the previous version, but now each value corresponds to the following properties:
ValueProperty

DEFAULT

use_all_dns_ips

USE_ALL_DNS_IPS

use_all_dns_ips

RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

resolve_canonical_bootstrap_servers_only

## Troubleshooting Publishing Null Values When the **Publish** operation receives null as input, the operation publishes an empty byte array. To avoid this issue, the connector enables you to set the `mule.kafka.publish.useNull` boolean system property. - When set to `true`, the connector publishes a message with null as a value. - When set to `false` or not set at all, the connector publishes a message with an empty byte array value. For further information on how to configure properties, refer to the [Use system properties](https://docs.mulesoft.com/mule-runtime/latest/mule-app-properties-system) ## Understand Commonly Thrown Exceptions Here is a list of commonly thrown exception messages and how to interpret them: - KAFKA:ALREADY\_COMMITTED A commit fails because the commit is already performed. - KAFKA:AUTHENTICATION\_ERROR(SECURITY) An authentication error is thrown. - KAFKA:AUTHORIZATION\_ERROR(CLIENT\_SECURITY) An authorization error occurred due to the client configuration or permissions. - KAFKA:COMMIT\_FAILED A commit fails with an unrecoverable error. - KAFKA:CONNECTIVITY Either a connection could not be established or there is a connectivity error. - KAFKA:ILLEGAL\_STATE The consumer is not subscribed to any topics, and no manually assigned partitions are set to consume from. - KAFKA:INPUT\_TOO\_LARGE The input is too large for the broker. - KAFKA:INVALID\_ACK\_MODE An acknowledgement (ACK) or negative acknowledgement (NACK) operation is run over a consumer that is not in \`MANUAL\` mode. - KAFKA:INVALID\_CONFIGURATION An invalid configuration value is set for the consumer or the producer. - KAFKA:INVALID\_CONNECTION The referenced connection is invalid. - KAFKA:INVALID\_INPUT The parameters set for the operation are incorrect. - KAFKA:INVALID\_OFFSET The partition number set for the operation is less than \`0\`, is above the number of partitions, or is an invalid value. - KAFKA:INVALID\_TOPIC The topic selected for the operation is invalid, and there is no automatic generation of said topic. This might also imply an invalid character in the topic name. - KAFKA:INVALID\_TOPIC\_PARTITION The topic partition combination selected for the operation is invalid, and there is no automatic generation of topics. This might also imply an invalid character in the topic name, or a non-existing partition. - KAFKA:NO\_POLL\_MADE No poll was made. - KAFKA:NOT\_FOUND For consume operations, the poll does not bring any records for the configured timeouts. - KAFKA:OUT\_OF\_RANGE There is no reset policy defined, and the offsets for these partitions are either larger or smaller than the range of offsets the server has for the given partition. - KAFKA:PREVIOUS\_ASSIGNATION subscribe() is called previously with topics or pattern without a subsequent call to unsubscribe(). - KAFKA:PRODUCER\_FENCED Another producer with the same \`transactional.id\` is active. - KAFKA:RETRY\_EXHAUSTED The maximum number of retries for the operation is reached. - KAFKA:SESSION\_NOT\_FOUND No associated session is found when executing an operation. - KAFKA:TIMEOUT A specific request takes longer than the configuration timeout values. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Apache Kafka Connector Reference](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-reference) - [Salesforce Help](https://help.salesforce.com) - [Salesforce Help](https://help.salesforce.com) - [Known Issues](https://help.salesforce.com/s/issues) --- product: Apache Kafka Connector version: 4.14 --- # Troubleshooting Apache Kafka Connector 4.14 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt To troubleshoot Anypoint Connector for Apache Kafka (Apache Kafka Connector), become familiar with the information about interpreting commonly thrown exception messages. ## Enable SSL Logging Enable SSL logs to investigate an issue related to TLS communications. Set your SSL log level depending on your required level of detail: 1. Enable only SSL log details: `javax.net.debug=ssl` 2. Enable only handshaking details. (The handshake protocol is a series of messages exchanged over the record protocol.) `javax.net.debug=ssl:handshake` 3. Enable handshaking details and SSL details at the same time: `javax.net.debug=ssl, handshake` 4. Enable the dumping of all details and traffic data: `javax.net.debug=all` > [!NOTE] This option is very verbose and under normal circumstances is not necessary. > [!WARNING] SSL logging results in a performance impact for HTTPS or other TLS connections. Enable > SSL logging only to troubleshoot a specific issue and do not enable it for extended periods. SSL > logging produces a significant amount of log messages and can inundate your log file if it is left > enabled and unattended. To enable SSL logs: Set the debug parameter as an argument in the runtime configuration for the application. 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and add `-M-Djavax.net.debug=ssl` in the **VM arguments** section. ## Enable Verbose Exception Logging Enable verbose exception logs to show a complete stack trace of the error instead of the default truncated output: 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and append the arguments in the **VM arguments** section to add the `mule.verbose.exceptions` property. For example: `-XX:PermSize=128M -XX:MaxPermSize=256M -Dmule.verbose.exceptions=true` ## Enable Verbose Logging To get a better understanding of why an application’s interaction with Kafka Connector is failing, temporarily enable verbose logging for the connector. Remember to always disable enhanced verbosity after troubleshooting, because it can affect your Mule application’s performance. To enable verbose logging in the configuration file: 1. Access Anypoint Studio and navigate to the **Package Explorer** view. 2. Open your application’s project name. 3. Open the `src/main/resources` folder. 4. Open the `log4j2.xml` file inside the folder. 5. Add an `` tag inside the `` tag: ```xml ... ... ``` 6. Save your application changes. 7. Click the project name in **Package Explorer** and then click **Run** > **Run As** > **Mule Application**. ## Enable FIPS Mode Enable FIPS mode by setting the `com.mulesoft.connectors.allow-registration-of-sasl-security-provider-in-fips=true` flag to true. This adds the `com.sun.security.sasl.Provider` security provider in the FIPS environment. ## Failed to Load SSL Keystore When the Kafka Connector configurations fail to load the required truststore files from the classpath, you receive the following error: ```text org.mule.runtime.api.connection.ConnectionException: Failed to load SSL keystore ``` The Kafka Connector configurations bundled with a dependency fail to establish connections even when the required truststore files are available in the classpath. To resolve this error: Add the required truststore files to the main application that uses the Kafka Connector configurations. ## Troubleshooting clientDNSLookUp Property Since Kafka Connector version 3.0.0, the default value for the **DNS Lookup** field was removed to keep the connector backwards compatible. Kafka Connector keeps the same values as the previous version, but now each value corresponds to the following properties:
ValueProperty

DEFAULT

use_all_dns_ips

USE_ALL_DNS_IPS

use_all_dns_ips

RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

resolve_canonical_bootstrap_servers_only

## Troubleshooting Publishing Null Values When the **Publish** operation receives null as input, the operation publishes an empty byte array. To avoid this issue, the connector enables you to set the `mule.kafka.publish.useNull` boolean system property. - When set to `true`, the connector publishes a message with null as a value. - When set to `false` or not set at all, the connector publishes a message with an empty byte array value. For further information on how to configure properties, refer to the [Use system properties](https://docs.mulesoft.com/mule-runtime/latest/mule-app-properties-system) ## Understand Commonly Thrown Exceptions Here is a list of commonly thrown exception messages and how to interpret them: - KAFKA:ALREADY\_COMMITTED A commit fails because the commit is already performed. - KAFKA:AUTHENTICATION\_ERROR(SECURITY) An authentication error is thrown. - KAFKA:AUTHORIZATION\_ERROR(CLIENT\_SECURITY) An authorization error occurred due to the client configuration or permissions. - KAFKA:COMMIT\_FAILED A commit fails with an unrecoverable error. - KAFKA:CONNECTIVITY Either a connection could not be established or there is a connectivity error. - KAFKA:ILLEGAL\_STATE The consumer is not subscribed to any topics, and no manually assigned partitions are set to consume from. - KAFKA:INPUT\_TOO\_LARGE The input is too large for the broker. - KAFKA:INVALID\_ACK\_MODE An acknowledgement (ACK) or negative acknowledgement (NACK) operation is run over a consumer that is not in \`MANUAL\` mode. - KAFKA:INVALID\_CONFIGURATION An invalid configuration value is set for the consumer or the producer. - KAFKA:INVALID\_CONNECTION The referenced connection is invalid. - KAFKA:INVALID\_INPUT The parameters set for the operation are incorrect. - KAFKA:INVALID\_OFFSET The partition number set for the operation is less than \`0\`, is above the number of partitions, or is an invalid value. - KAFKA:INVALID\_TOPIC The topic selected for the operation is invalid, and there is no automatic generation of said topic. This might also imply an invalid character in the topic name. - KAFKA:INVALID\_TOPIC\_PARTITION The topic partition combination selected for the operation is invalid, and there is no automatic generation of topics. This might also imply an invalid character in the topic name, or a non-existing partition. - KAFKA:NO\_POLL\_MADE No poll was made. - KAFKA:NOT\_FOUND For consume operations, the poll does not bring any records for the configured timeouts. - KAFKA:OUT\_OF\_RANGE There is no reset policy defined, and the offsets for these partitions are either larger or smaller than the range of offsets the server has for the given partition. - KAFKA:PREVIOUS\_ASSIGNATION subscribe() is called previously with topics or pattern without a subsequent call to unsubscribe(). - KAFKA:PRODUCER\_FENCED Another producer with the same \`transactional.id\` is active. - KAFKA:RETRY\_EXHAUSTED The maximum number of retries for the operation is reached. - KAFKA:SESSION\_NOT\_FOUND No associated session is found when executing an operation. - KAFKA:TIMEOUT A specific request takes longer than the configuration timeout values. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Apache Kafka Connector Reference](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-reference) - [Salesforce Help](https://help.salesforce.com) - [Salesforce Help](https://help.salesforce.com) - [Known Issues](https://help.salesforce.com/s/issues) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Troubleshooting Apache Kafka Connector 4.15 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt To troubleshoot Anypoint Connector for Apache Kafka (Apache Kafka Connector), become familiar with the information about interpreting commonly thrown exception messages. ## Enable SSL Logging Enable SSL logs to investigate an issue related to TLS communications. Set your SSL log level depending on your required level of detail: 1. Enable only SSL log details: `javax.net.debug=ssl` 2. Enable only handshaking details. (The handshake protocol is a series of messages exchanged over the record protocol.) `javax.net.debug=ssl:handshake` 3. Enable handshaking details and SSL details at the same time: `javax.net.debug=ssl, handshake` 4. Enable the dumping of all details and traffic data: `javax.net.debug=all` > [!NOTE] This option is very verbose and under normal circumstances is not necessary. > [!WARNING] SSL logging results in a performance impact for HTTPS or other TLS connections. Enable > SSL logging only to troubleshoot a specific issue and do not enable it for extended periods. SSL > logging produces a significant amount of log messages and can inundate your log file if it is left > enabled and unattended. To enable SSL logs: Set the debug parameter as an argument in the runtime configuration for the application. 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and add `-M-Djavax.net.debug=ssl` in the **VM arguments** section. ## Enable Verbose Exception Logging Enable verbose exception logs to show a complete stack trace of the error instead of the default truncated output: 1. In Studio, right-click on the project and select **Run > Run Configurations**. 2. Go to the **Arguments** tab and append the arguments in the **VM arguments** section to add the `mule.verbose.exceptions` property. For example: `-XX:PermSize=128M -XX:MaxPermSize=256M -Dmule.verbose.exceptions=true` ## Enable Verbose Logging To get a better understanding of why an application’s interaction with Kafka Connector is failing, temporarily enable verbose logging for the connector. Remember to always disable enhanced verbosity after troubleshooting, because it can affect your Mule application’s performance. To enable verbose logging in the configuration file: 1. Access Anypoint Studio and navigate to the **Package Explorer** view. 2. Open your application’s project name. 3. Open the `src/main/resources` folder. 4. Open the `log4j2.xml` file inside the folder. 5. Add an `` tag inside the `` tag: ```xml ... ... ``` 6. Save your application changes. 7. Click the project name in **Package Explorer** and then click **Run** > **Run As** > **Mule Application**. ## Enable FIPS Mode Enable FIPS mode by setting the `com.mulesoft.connectors.allow-registration-of-sasl-security-provider-in-fips=true` flag to true. This adds the `com.sun.security.sasl.Provider` security provider in the FIPS environment. ## Failed to Load SSL Keystore When the Kafka Connector configurations fail to load the required truststore files from the classpath, you receive the following error: ```text org.mule.runtime.api.connection.ConnectionException: Failed to load SSL keystore ``` The Kafka Connector configurations bundled with a dependency fail to establish connections even when the required truststore files are available in the classpath. To resolve this error: Add the required truststore files to the main application that uses the Kafka Connector configurations. ## Troubleshooting clientDNSLookUp Property Since Kafka Connector version 3.0.0, the default value for the **DNS Lookup** field was removed to keep the connector backwards compatible. Kafka Connector keeps the same values as the previous version, but now each value corresponds to the following properties:
ValueProperty

DEFAULT

use_all_dns_ips

USE_ALL_DNS_IPS

use_all_dns_ips

RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY

resolve_canonical_bootstrap_servers_only

## Troubleshooting Publishing Null Values When the **Publish** operation receives null as input, the operation publishes an empty byte array. To avoid this issue, the connector enables you to set the `mule.kafka.publish.useNull` boolean system property. - When set to `true`, the connector publishes a message with null as a value. - When set to `false` or not set at all, the connector publishes a message with an empty byte array value. For further information on how to configure properties, refer to the [Use system properties](https://docs.mulesoft.com/mule-runtime/latest/mule-app-properties-system) ## Understand Commonly Thrown Exceptions Here is a list of commonly thrown exception messages and how to interpret them: - KAFKA:ALREADY\_COMMITTED A commit fails because the commit is already performed. - KAFKA:AUTHENTICATION\_ERROR(SECURITY) An authentication error is thrown. - KAFKA:AUTHORIZATION\_ERROR(CLIENT\_SECURITY) An authorization error occurred due to the client configuration or permissions. - KAFKA:COMMIT\_FAILED A commit fails with an unrecoverable error. - KAFKA:CONNECTIVITY Either a connection could not be established or there is a connectivity error. - KAFKA:ILLEGAL\_STATE The consumer is not subscribed to any topics, and no manually assigned partitions are set to consume from. - KAFKA:INPUT\_TOO\_LARGE The input is too large for the broker. - KAFKA:INVALID\_ACK\_MODE An acknowledgement (ACK) or negative acknowledgement (NACK) operation is run over a consumer that is not in \`MANUAL\` mode. - KAFKA:INVALID\_CONFIGURATION An invalid configuration value is set for the consumer or the producer. - KAFKA:INVALID\_CONNECTION The referenced connection is invalid. - KAFKA:INVALID\_INPUT The parameters set for the operation are incorrect. - KAFKA:INVALID\_OFFSET The partition number set for the operation is less than \`0\`, is above the number of partitions, or is an invalid value. - KAFKA:INVALID\_TOPIC The topic selected for the operation is invalid, and there is no automatic generation of said topic. This might also imply an invalid character in the topic name. - KAFKA:INVALID\_TOPIC\_PARTITION The topic partition combination selected for the operation is invalid, and there is no automatic generation of topics. This might also imply an invalid character in the topic name, or a non-existing partition. - KAFKA:NO\_POLL\_MADE No poll was made. - KAFKA:NOT\_FOUND For consume operations, the poll does not bring any records for the configured timeouts. - KAFKA:OUT\_OF\_RANGE There is no reset policy defined, and the offsets for these partitions are either larger or smaller than the range of offsets the server has for the given partition. - KAFKA:PREVIOUS\_ASSIGNATION subscribe() is called previously with topics or pattern without a subsequent call to unsubscribe(). - KAFKA:PRODUCER\_FENCED Another producer with the same \`transactional.id\` is active. - KAFKA:RETRY\_EXHAUSTED The maximum number of retries for the operation is reached. - KAFKA:SESSION\_NOT\_FOUND No associated session is found when executing an operation. - KAFKA:TIMEOUT A specific request takes longer than the configuration timeout values. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Apache Kafka Connector Reference](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-reference) - [Salesforce Help](https://help.salesforce.com) - [Salesforce Help](https://help.salesforce.com) - [Known Issues](https://help.salesforce.com/s/issues) --- product: Apache Kafka Connector version: 4.10 --- # Upgrading and Migrating Apache Kafka Connector to 4.x > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Upgrade Anypoint Connector for Apache Kafka (Apache Kafka Connector) to version 4.x. ## Supported Upgrade Paths
From VersionTo Version

3.2.x

4.x

## Changes in This Release Apache Kafka Connector 4.x contains a lot of changes to improve efficiency, including new operations to enable you to better understand how to use the connector. All of the changes included in this version break backward compatibility. This release contains the following changes: - Four new operations with new parameters were added: - Commit This operation commits the offsets associated with a message or batch of messages consumed in a Message Listener (Source) or in a Batch Message Listener (Source). This operation works only with MANUAL Ack mode. - Consume This operation is used for receiving messages from one or more Kafka topics. The Consume operation works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation. - Publish This operation enables sending messages to a Kafka topic. It is non-blocking, and its behavior depends on specific values of the connection parameters. This operation supports transactions, so a transaction ID is randomly generated for the Kafka Producer. - Seek This operation is used for setting the current offset of the consumer for a given topic and partition to the provided offset value. See the [New Operations](#new_operations) table. - The Message Listener replaces the Message Consumer source. See the [Changed Sources](#changed_sources) table. - Added a Batch Message Listener source, which works in a similar way as the Message Listener with the difference that it processes the list of messages instead of a single message at a time. The message list obtained in the poll is handled by the Mule flow as a single event. See the [New Sources](#new_sources) table for information about the new parameters. - Added different acknowledgment modes in the consumer configuration, including: - AUTO Mule acknowledges the message only if the flow is finished successfully. - MANUAL The user must manually acknowledge the message within the flow. - DUPS\_OK Same as the AUTO mode, but the commit is made asynchronously, which can result in duplicate records. - IMMEDIATE Mule automatically acknowledges the message upon receipt. - There is now one plaintext connection type that supports transactions. In version 3.x of the connector, you could add a customizable set of configuration properties related to connections. In version 4.x of the connector, this feature was removed. The connector now has a set of connection types for which you can set properties. ## Changed Sources
Kafka SourceEarlier versionParameters

Message Listener

Message consumer

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## New Operations This release contains the following new operations:
Kafka OperationDescriptionParameters

Commit

Commits the offsets associated to a message or batch of messages consumed in a Message Listener.

Consumer commit key. The consumer commit key to use to commit the message.

Consume

Enables receiving messages from one or more Kafka topics. It works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation.

You must specify the timeout parameter, or the flow will be blocked.
  • Consumption timeout

  • Timeout time unit

Publish

Enables sending messages to a Kafka topic, and is non-blocking. Its behavior depends on specific values of the connection parameters. This operation supports transactions, which means that a transaction ID is randomly generated for the Kafka producer and handled in the connection.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Key

  • Message

  • Headers

Seek

Sets the current offset of the consumer for the given topic and partition to the provided offset value.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Offset (The offset to seek to)

> [!NOTE] With the exception of the Commit operation, the flow will be blocked if the `timeout` > parameter is not specified. ## New Sources
Kafka OperationDescriptionParameters

Batch Message Listener

Works in the same way as the Message Listener except that it processes the list of messages instead of processing a single message at a time. The message list that was obtained in the poll is handled by a flow as a single event, so the handling of concurrency is simpler than in the simple Message Listener. In other words, a commit of the messages happens for all the messages as a whole by calling the commit on the Kafka consumer.

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## Upgrade Prerequisites Before you perform the upgrade, you must create a backup of your files, data, and configurations in case you need to revert to the previous version. ## Upgrade Steps Follow these steps to perform the upgrade to Apache Kafka Connector 4.0.x: 1. In Studio, create a Mule project. 2. In the Mule Palette view, click **Search in Exchange**. 3. In Add Modules to Project, type 'Apache Kafka' in the search field. 4. In Available modules, select **Apache Kafka Connector** and then click **Add**. 5. Click **Finish**. Anypoint Studio upgrades the connector automatically. 6. Verify that the `kafka-connector` dependency version is `4.0.0` in the pom.xml. ## Post Upgrade Steps After you install the latest version of the connector, follow these steps to complete the upgrade: 1. In Anypoint Studio, verify that there are no errors in the **Problems** or **Console** views. 2. Check the project pom.xml and verify that there are no problems. 3. Test the connection and verify that the operations work. ## Troubleshooting If there are problems with caching the parameters and caching the metadata, try restarting Anypoint Studio. ### Reverting the Upgrade If it is necessary to revert to the previous version of Apache Kafka Connector, in Anypoint Studio, change the `kafka-connector` dependency version `4.0.0` in the project’s pom.xml to the previous version. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.11 --- # Upgrading and Migrating Apache Kafka Connector to 4.x > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Upgrade Anypoint Connector for Apache Kafka (Apache Kafka Connector) to version 4.x. ## Supported Upgrade Paths
From VersionTo Version

3.2.x

4.x

## Changes in This Release Apache Kafka Connector 4.x contains a lot of changes to improve efficiency, including new operations to enable you to better understand how to use the connector. All of the changes included in this version break backward compatibility. This release contains the following changes: - Four new operations with new parameters were added: - Commit This operation commits the offsets associated with a message or batch of messages consumed in a Message Listener (Source) or in a Batch Message Listener (Source). This operation works only with MANUAL Ack mode. - Consume This operation is used for receiving messages from one or more Kafka topics. The Consume operation works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation. - Publish This operation enables sending messages to a Kafka topic. It is non-blocking, and its behavior depends on specific values of the connection parameters. This operation supports transactions, so a transaction ID is randomly generated for the Kafka Producer. - Seek This operation is used for setting the current offset of the consumer for a given topic and partition to the provided offset value. See the [New Operations](#new_operations) table. - The Message Listener replaces the Message Consumer source. See the [Changed Sources](#changed_sources) table. - Added a Batch Message Listener source, which works in a similar way as the Message Listener with the difference that it processes the list of messages instead of a single message at a time. The message list obtained in the poll is handled by the Mule flow as a single event. See the [New Sources](#new_sources) table for information about the new parameters. - Added different acknowledgment modes in the consumer configuration, including: - AUTO Mule acknowledges the message only if the flow is finished successfully. - MANUAL The user must manually acknowledge the message within the flow. - DUPS\_OK Same as the AUTO mode, but the commit is made asynchronously, which can result in duplicate records. - IMMEDIATE Mule automatically acknowledges the message upon receipt. - There is now one plaintext connection type that supports transactions. In version 3.x of the connector, you could add a customizable set of configuration properties related to connections. In version 4.x of the connector, this feature was removed. The connector now has a set of connection types for which you can set properties. ## Changed Sources
Kafka SourceEarlier versionParameters

Message Listener

Message consumer

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## New Operations This release contains the following new operations:
Kafka OperationDescriptionParameters

Commit

Commits the offsets associated to a message or batch of messages consumed in a Message Listener.

Consumer commit key. The consumer commit key to use to commit the message.

Consume

Enables receiving messages from one or more Kafka topics. It works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation.

You must specify the timeout parameter, or the flow will be blocked.
  • Consumption timeout

  • Timeout time unit

Publish

Enables sending messages to a Kafka topic, and is non-blocking. Its behavior depends on specific values of the connection parameters. This operation supports transactions, which means that a transaction ID is randomly generated for the Kafka producer and handled in the connection.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Key

  • Message

  • Headers

Seek

Sets the current offset of the consumer for the given topic and partition to the provided offset value.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Offset (The offset to seek to)

> [!NOTE] With the exception of the Commit operation, the flow will be blocked if the `timeout` > parameter is not specified. ## New Sources
Kafka OperationDescriptionParameters

Batch Message Listener

Works in the same way as the Message Listener except that it processes the list of messages instead of processing a single message at a time. The message list that was obtained in the poll is handled by a flow as a single event, so the handling of concurrency is simpler than in the simple Message Listener. In other words, a commit of the messages happens for all the messages as a whole by calling the commit on the Kafka consumer.

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## Upgrade Prerequisites Before you perform the upgrade, you must create a backup of your files, data, and configurations in case you need to revert to the previous version. ## Upgrade Steps Follow these steps to perform the upgrade to Apache Kafka Connector 4.0.x: 1. In Studio, create a Mule project. 2. In the Mule Palette view, click **Search in Exchange**. 3. In Add Modules to Project, type 'Apache Kafka' in the search field. 4. In Available modules, select **Apache Kafka Connector** and then click **Add**. 5. Click **Finish**. Anypoint Studio upgrades the connector automatically. 6. Verify that the `kafka-connector` dependency version is `4.0.0` in the pom.xml. ## Post Upgrade Steps After you install the latest version of the connector, follow these steps to complete the upgrade: 1. In Anypoint Studio, verify that there are no errors in the **Problems** or **Console** views. 2. Check the project pom.xml and verify that there are no problems. 3. Test the connection and verify that the operations work. ## Troubleshooting If there are problems with caching the parameters and caching the metadata, try restarting Anypoint Studio. ### Reverting the Upgrade If it is necessary to revert to the previous version of Apache Kafka Connector, in Anypoint Studio, change the `kafka-connector` dependency version `4.0.0` in the project’s pom.xml to the previous version. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.12 --- # Upgrading and Migrating Apache Kafka Connector to 4.x > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Upgrade Anypoint Connector for Apache Kafka (Apache Kafka Connector) to version 4.x. ## Supported Upgrade Paths
From VersionTo Version

3.2.x

4.x

## Changes in This Release Apache Kafka Connector 4.x contains a lot of changes to improve efficiency, including new operations to enable you to better understand how to use the connector. All of the changes included in this version break backward compatibility. This release contains the following changes: - Four new operations with new parameters were added: - Commit This operation commits the offsets associated with a message or batch of messages consumed in a Message Listener (Source) or in a Batch Message Listener (Source). This operation works only with MANUAL Ack mode. - Consume This operation is used for receiving messages from one or more Kafka topics. The Consume operation works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation. - Publish This operation enables sending messages to a Kafka topic. It is non-blocking, and its behavior depends on specific values of the connection parameters. This operation supports transactions, so a transaction ID is randomly generated for the Kafka Producer. - Seek This operation is used for setting the current offset of the consumer for a given topic and partition to the provided offset value. See the [New Operations](#new_operations) table. - The Message Listener replaces the Message Consumer source. See the [Changed Sources](#changed_sources) table. - Added a Batch Message Listener source, which works in a similar way as the Message Listener with the difference that it processes the list of messages instead of a single message at a time. The message list obtained in the poll is handled by the Mule flow as a single event. See the [New Sources](#new_sources) table for information about the new parameters. - Added different acknowledgment modes in the consumer configuration, including: - AUTO Mule acknowledges the message only if the flow is finished successfully. - MANUAL The user must manually acknowledge the message within the flow. - DUPS\_OK Same as the AUTO mode, but the commit is made asynchronously, which can result in duplicate records. - IMMEDIATE Mule automatically acknowledges the message upon receipt. - There is now one plaintext connection type that supports transactions. In version 3.x of the connector, you could add a customizable set of configuration properties related to connections. In version 4.x of the connector, this feature was removed. The connector now has a set of connection types for which you can set properties. ## Changed Sources
Kafka SourceEarlier versionParameters

Message Listener

Message consumer

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## New Operations This release contains the following new operations:
Kafka OperationDescriptionParameters

Commit

Commits the offsets associated to a message or batch of messages consumed in a Message Listener.

Consumer commit key. The consumer commit key to use to commit the message.

Consume

Enables receiving messages from one or more Kafka topics. It works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation.

You must specify the timeout parameter, or the flow will be blocked.
  • Consumption timeout

  • Timeout time unit

Publish

Enables sending messages to a Kafka topic, and is non-blocking. Its behavior depends on specific values of the connection parameters. This operation supports transactions, which means that a transaction ID is randomly generated for the Kafka producer and handled in the connection.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Key

  • Message

  • Headers

Seek

Sets the current offset of the consumer for the given topic and partition to the provided offset value.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Offset (The offset to seek to)

> [!NOTE] With the exception of the Commit operation, the flow will be blocked if the `timeout` > parameter is not specified. ## New Sources
Kafka OperationDescriptionParameters

Batch Message Listener

Works in the same way as the Message Listener except that it processes the list of messages instead of processing a single message at a time. The message list that was obtained in the poll is handled by a flow as a single event, so the handling of concurrency is simpler than in the simple Message Listener. In other words, a commit of the messages happens for all the messages as a whole by calling the commit on the Kafka consumer.

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## Upgrade Prerequisites Before you perform the upgrade, you must create a backup of your files, data, and configurations in case you need to revert to the previous version. ## Upgrade Steps Follow these steps to perform the upgrade to Apache Kafka Connector 4.0.x: 1. In Studio, create a Mule project. 2. In the Mule Palette view, click **Search in Exchange**. 3. In Add Modules to Project, type 'Apache Kafka' in the search field. 4. In Available modules, select **Apache Kafka Connector** and then click **Add**. 5. Click **Finish**. Anypoint Studio upgrades the connector automatically. 6. Verify that the `kafka-connector` dependency version is `4.0.0` in the pom.xml. ## Post Upgrade Steps After you install the latest version of the connector, follow these steps to complete the upgrade: 1. In Anypoint Studio, verify that there are no errors in the **Problems** or **Console** views. 2. Check the project pom.xml and verify that there are no problems. 3. Test the connection and verify that the operations work. ## Troubleshooting If there are problems with caching the parameters and caching the metadata, try restarting Anypoint Studio. ### Reverting the Upgrade If it is necessary to revert to the previous version of Apache Kafka Connector, in Anypoint Studio, change the `kafka-connector` dependency version `4.0.0` in the project’s pom.xml to the previous version. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.13 --- # Upgrading and Migrating Apache Kafka Connector to 4.x > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Upgrade Anypoint Connector for Apache Kafka (Apache Kafka Connector) to version 4.x. ## Supported Upgrade Paths
From VersionTo Version

3.2.x

4.x

## Changes in This Release Apache Kafka Connector 4.x contains a lot of changes to improve efficiency, including new operations to enable you to better understand how to use the connector. All of the changes included in this version break backward compatibility. This release contains the following changes: - Four new operations with new parameters were added: - Commit This operation commits the offsets associated with a message or batch of messages consumed in a Message Listener (Source) or in a Batch Message Listener (Source). This operation works only with MANUAL Ack mode. - Consume This operation is used for receiving messages from one or more Kafka topics. The Consume operation works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation. - Publish This operation enables sending messages to a Kafka topic. It is non-blocking, and its behavior depends on specific values of the connection parameters. This operation supports transactions, so a transaction ID is randomly generated for the Kafka Producer. - Seek This operation is used for setting the current offset of the consumer for a given topic and partition to the provided offset value. See the [New Operations](#new_operations) table. - The Message Listener replaces the Message Consumer source. See the [Changed Sources](#changed_sources) table. - Added a Batch Message Listener source, which works in a similar way as the Message Listener with the difference that it processes the list of messages instead of a single message at a time. The message list obtained in the poll is handled by the Mule flow as a single event. See the [New Sources](#new_sources) table for information about the new parameters. - Added different acknowledgment modes in the consumer configuration, including: - AUTO Mule acknowledges the message only if the flow is finished successfully. - MANUAL The user must manually acknowledge the message within the flow. - DUPS\_OK Same as the AUTO mode, but the commit is made asynchronously, which can result in duplicate records. - IMMEDIATE Mule automatically acknowledges the message upon receipt. - There is now one plaintext connection type that supports transactions. In version 3.x of the connector, you could add a customizable set of configuration properties related to connections. In version 4.x of the connector, this feature was removed. The connector now has a set of connection types for which you can set properties. ## Changed Sources
Kafka SourceEarlier versionParameters

Message Listener

Message consumer

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## New Operations This release contains the following new operations:
Kafka OperationDescriptionParameters

Commit

Commits the offsets associated to a message or batch of messages consumed in a Message Listener.

Consumer commit key. The consumer commit key to use to commit the message.

Consume

Enables receiving messages from one or more Kafka topics. It works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation.

You must specify the timeout parameter, or the flow will be blocked.
  • Consumption timeout

  • Timeout time unit

Publish

Enables sending messages to a Kafka topic, and is non-blocking. Its behavior depends on specific values of the connection parameters. This operation supports transactions, which means that a transaction ID is randomly generated for the Kafka producer and handled in the connection.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Key

  • Message

  • Headers

Seek

Sets the current offset of the consumer for the given topic and partition to the provided offset value.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Offset (The offset to seek to)

> [!NOTE] With the exception of the Commit operation, the flow will be blocked if the `timeout` > parameter is not specified. ## New Sources
Kafka OperationDescriptionParameters

Batch Message Listener

Works in the same way as the Message Listener except that it processes the list of messages instead of processing a single message at a time. The message list that was obtained in the poll is handled by a flow as a single event, so the handling of concurrency is simpler than in the simple Message Listener. In other words, a commit of the messages happens for all the messages as a whole by calling the commit on the Kafka consumer.

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## Upgrade Prerequisites Before you perform the upgrade, you must create a backup of your files, data, and configurations in case you need to revert to the previous version. ## Upgrade Steps Follow these steps to perform the upgrade to Apache Kafka Connector 4.0.x: 1. In Studio, create a Mule project. 2. In the Mule Palette view, click **Search in Exchange**. 3. In Add Modules to Project, type 'Apache Kafka' in the search field. 4. In Available modules, select **Apache Kafka Connector** and then click **Add**. 5. Click **Finish**. Anypoint Studio upgrades the connector automatically. 6. Verify that the `kafka-connector` dependency version is `4.0.0` in the pom.xml. ## Post Upgrade Steps After you install the latest version of the connector, follow these steps to complete the upgrade: 1. In Anypoint Studio, verify that there are no errors in the **Problems** or **Console** views. 2. Check the project pom.xml and verify that there are no problems. 3. Test the connection and verify that the operations work. ## Troubleshooting If there are problems with caching the parameters and caching the metadata, try restarting Anypoint Studio. ### Reverting the Upgrade If it is necessary to revert to the previous version of Apache Kafka Connector, in Anypoint Studio, change the `kafka-connector` dependency version `4.0.0` in the project’s pom.xml to the previous version. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.14 --- # Upgrading and Migrating Apache Kafka Connector to 4.x > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Upgrade Anypoint Connector for Apache Kafka (Apache Kafka Connector) to version 4.x. ## Supported Upgrade Paths
From VersionTo Version

3.2.x

4.x

## Changes in This Release Apache Kafka Connector 4.x contains a lot of changes to improve efficiency, including new operations to enable you to better understand how to use the connector. All of the changes included in this version break backward compatibility. This release contains the following changes: - Four new operations with new parameters were added: - Commit This operation commits the offsets associated with a message or batch of messages consumed in a Message Listener (Source) or in a Batch Message Listener (Source). This operation works only with MANUAL Ack mode. - Consume This operation is used for receiving messages from one or more Kafka topics. The Consume operation works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation. - Publish This operation enables sending messages to a Kafka topic. It is non-blocking, and its behavior depends on specific values of the connection parameters. This operation supports transactions, so a transaction ID is randomly generated for the Kafka Producer. - Seek This operation is used for setting the current offset of the consumer for a given topic and partition to the provided offset value. See the [New Operations](#new_operations) table. - The Message Listener replaces the Message Consumer source. See the [Changed Sources](#changed_sources) table. - Added a Batch Message Listener source, which works in a similar way as the Message Listener with the difference that it processes the list of messages instead of a single message at a time. The message list obtained in the poll is handled by the Mule flow as a single event. See the [New Sources](#new_sources) table for information about the new parameters. - Added different acknowledgment modes in the consumer configuration, including: - AUTO Mule acknowledges the message only if the flow is finished successfully. - MANUAL The user must manually acknowledge the message within the flow. - DUPS\_OK Same as the AUTO mode, but the commit is made asynchronously, which can result in duplicate records. - IMMEDIATE Mule automatically acknowledges the message upon receipt. - There is now one plaintext connection type that supports transactions. In version 3.x of the connector, you could add a customizable set of configuration properties related to connections. In version 4.x of the connector, this feature was removed. The connector now has a set of connection types for which you can set properties. ## Changed Sources
Kafka SourceEarlier versionParameters

Message Listener

Message consumer

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## New Operations This release contains the following new operations:
Kafka OperationDescriptionParameters

Commit

Commits the offsets associated to a message or batch of messages consumed in a Message Listener.

Consumer commit key. The consumer commit key to use to commit the message.

Consume

Enables receiving messages from one or more Kafka topics. It works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation.

You must specify the timeout parameter, or the flow will be blocked.
  • Consumption timeout

  • Timeout time unit

Publish

Enables sending messages to a Kafka topic, and is non-blocking. Its behavior depends on specific values of the connection parameters. This operation supports transactions, which means that a transaction ID is randomly generated for the Kafka producer and handled in the connection.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Key

  • Message

  • Headers

Seek

Sets the current offset of the consumer for the given topic and partition to the provided offset value.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Offset (The offset to seek to)

> [!NOTE] With the exception of the Commit operation, the flow will be blocked if the `timeout` > parameter is not specified. ## New Sources
Kafka OperationDescriptionParameters

Batch Message Listener

Works in the same way as the Message Listener except that it processes the list of messages instead of processing a single message at a time. The message list that was obtained in the poll is handled by a flow as a single event, so the handling of concurrency is simpler than in the simple Message Listener. In other words, a commit of the messages happens for all the messages as a whole by calling the commit on the Kafka consumer.

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## Upgrade Prerequisites Before you perform the upgrade, you must create a backup of your files, data, and configurations in case you need to revert to the previous version. ## Upgrade Steps Follow these steps to perform the upgrade to Apache Kafka Connector 4.0.x: 1. In Studio, create a Mule project. 2. In the Mule Palette view, click **Search in Exchange**. 3. In Add Modules to Project, type 'Apache Kafka' in the search field. 4. In Available modules, select **Apache Kafka Connector** and then click **Add**. 5. Click **Finish**. Anypoint Studio upgrades the connector automatically. 6. Verify that the `kafka-connector` dependency version is `4.0.0` in the pom.xml. ## Post Upgrade Steps After you install the latest version of the connector, follow these steps to complete the upgrade: 1. In Anypoint Studio, verify that there are no errors in the **Problems** or **Console** views. 2. Check the project pom.xml and verify that there are no problems. 3. Test the connection and verify that the operations work. ## Troubleshooting If there are problems with caching the parameters and caching the metadata, try restarting Anypoint Studio. ### Reverting the Upgrade If it is necessary to revert to the previous version of Apache Kafka Connector, in Anypoint Studio, change the `kafka-connector` dependency version `4.0.0` in the project’s pom.xml to the previous version. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Upgrading and Migrating Apache Kafka Connector to 4.x > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Upgrade Anypoint Connector for Apache Kafka (Apache Kafka Connector) to version 4.x. ## Supported Upgrade Paths
From VersionTo Version

3.2.x

4.x

## Changes in This Release Apache Kafka Connector 4.x contains a lot of changes to improve efficiency, including new operations to enable you to better understand how to use the connector. All of the changes included in this version break backward compatibility. This release contains the following changes: - Four new operations with new parameters were added: - Commit This operation commits the offsets associated with a message or batch of messages consumed in a Message Listener (Source) or in a Batch Message Listener (Source). This operation works only with MANUAL Ack mode. - Consume This operation is used for receiving messages from one or more Kafka topics. The Consume operation works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation. - Publish This operation enables sending messages to a Kafka topic. It is non-blocking, and its behavior depends on specific values of the connection parameters. This operation supports transactions, so a transaction ID is randomly generated for the Kafka Producer. - Seek This operation is used for setting the current offset of the consumer for a given topic and partition to the provided offset value. See the [New Operations](#new_operations) table. - The Message Listener replaces the Message Consumer source. See the [Changed Sources](#changed_sources) table. - Added a Batch Message Listener source, which works in a similar way as the Message Listener with the difference that it processes the list of messages instead of a single message at a time. The message list obtained in the poll is handled by the Mule flow as a single event. See the [New Sources](#new_sources) table for information about the new parameters. - Added different acknowledgment modes in the consumer configuration, including: - AUTO Mule acknowledges the message only if the flow is finished successfully. - MANUAL The user must manually acknowledge the message within the flow. - DUPS\_OK Same as the AUTO mode, but the commit is made asynchronously, which can result in duplicate records. - IMMEDIATE Mule automatically acknowledges the message upon receipt. - There is now one plaintext connection type that supports transactions. In version 3.x of the connector, you could add a customizable set of configuration properties related to connections. In version 4.x of the connector, this feature was removed. The connector now has a set of connection types for which you can set properties. ## Changed Sources
Kafka SourceEarlier versionParameters

Message Listener

Message consumer

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## New Operations This release contains the following new operations:
Kafka OperationDescriptionParameters

Commit

Commits the offsets associated to a message or batch of messages consumed in a Message Listener.

Consumer commit key. The consumer commit key to use to commit the message.

Consume

Enables receiving messages from one or more Kafka topics. It works similarly to the Message Listener source, so all the operations that apply to the Message Listener source also apply to this operation.

You must specify the timeout parameter, or the flow will be blocked.
  • Consumption timeout

  • Timeout time unit

Publish

Enables sending messages to a Kafka topic, and is non-blocking. Its behavior depends on specific values of the connection parameters. This operation supports transactions, which means that a transaction ID is randomly generated for the Kafka producer and handled in the connection.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Key

  • Message

  • Headers

Seek

Sets the current offset of the consumer for the given topic and partition to the provided offset value.

You must specify the timeout parameter, or the flow will be blocked.
  • Topic (The topic name)

  • Partition (The partition name)

  • Offset (The offset to seek to)

> [!NOTE] With the exception of the Commit operation, the flow will be blocked if the `timeout` > parameter is not specified. ## New Sources
Kafka OperationDescriptionParameters

Batch Message Listener

Works in the same way as the Message Listener except that it processes the list of messages instead of processing a single message at a time. The message list that was obtained in the poll is handled by a flow as a single event, so the handling of concurrency is simpler than in the simple Message Listener. In other words, a commit of the messages happens for all the messages as a whole by calling the commit on the Kafka consumer.

  • Poll timeout

  • Poll timeout time unit

  • Acknowledgment mode

  • Number of parallel consumers

## Upgrade Prerequisites Before you perform the upgrade, you must create a backup of your files, data, and configurations in case you need to revert to the previous version. ## Upgrade Steps Follow these steps to perform the upgrade to Apache Kafka Connector 4.0.x: 1. In Studio, create a Mule project. 2. In the Mule Palette view, click **Search in Exchange**. 3. In Add Modules to Project, type 'Apache Kafka' in the search field. 4. In Available modules, select **Apache Kafka Connector** and then click **Add**. 5. Click **Finish**. Anypoint Studio upgrades the connector automatically. 6. Verify that the `kafka-connector` dependency version is `4.0.0` in the pom.xml. ## Post Upgrade Steps After you install the latest version of the connector, follow these steps to complete the upgrade: 1. In Anypoint Studio, verify that there are no errors in the **Problems** or **Console** views. 2. Check the project pom.xml and verify that there are no problems. 3. Test the connection and verify that the operations work. ## Troubleshooting If there are problems with caching the parameters and caching the metadata, try restarting Anypoint Studio. ### Reverting the Upgrade If it is necessary to revert to the previous version of Apache Kafka Connector, in Anypoint Studio, change the `kafka-connector` dependency version `4.0.0` in the project’s pom.xml to the previous version. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.10 --- # Using Anypoint Studio to Configure Apache Kafka 4.10 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt In Anypoint Studio, add Anypoint Connector for Apache Kafka (Apache Kafka Connector) to a Mule project, configure the connection to the Kafka cluster, and configure an input source for the connector. To configure a connector in Anypoint Studio: 1. Add the connector to a Mule project. 2. Configure the connector. 3. Configure an input source for the connector. ## Add the Connector in Studio 1. In Studio, create a Mule Project. 2. In the Mule Palette, click **(X) Search in Exchange**. 3. In **Add Modules to Project**, type the name of the connector in the search field. 4. Click the connector name in **Available modules**. 5. Click **Add**. 6. Click **Finish**. ## Add the Connector Using Exchange 1. In Studio, create a Mule Project. 2. Click the Exchange **(X)** icon in the upper-left of the Studio task bar. 3. In Exchange, click **Login** and supply your Anypoint Platform username and password. 4. In Exchange, select **All assets** and search for `Apache Kafka`. 5. Select the connector and click **Add to project**. 6. Follow the prompts to add the connector to the Mule project. ## Consumer Configuration You can create a consumer configuration global element to reference from Apache Kafka Connector. This enables you to apply configuration details to multiple local elements in the flow. 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Consumer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:
Connection TypeField NameDescription

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka consumer clients.

Bean reference

URLs that the consumer can use to connect to the Kafka cluster

Group ID

Default group ID for all the Kafka consumers that use this configuration

Consumer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kafka configuration file (krb5.conf)

File that contains Kerberos configuration information

Consumer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Consumer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Consumer SASL/PLAIN

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Consumer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Producer Configuration Create a producer configuration global element: 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Producer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:

Connection Type

Field Name

Description

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka producer clients.

Bean reference

URLs that the producer can use to connect to the Kafka cluster

Producer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kerberos configuration file (krb5.conf)

File that contains Kerberos configuration information

Producer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Producer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Producer SASL/PLAIN

Username

Username with which to log in to Kafa

Password

Password with which to log in to Kafka

Producer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Configure TLS To enable TLS for your app: 1. Click the **TLS** tab to configure the truststore and keystore: - **Trust Store Configuration** - Path Location of the truststore file. - Password Password for the truststore file. - Type File format of the truststore file. - Algorithm Algorithm the truststore uses. - Insecure Boolean that determines whether or not to validate the truststore. If set to `true`, no validation occurs. The default value is `false`. - **Key Store Configuration** - Type Optionally specify the file format of the keystore file. The default value is `JKS`. - Path Location of the keystore file. This is optional and can be used for two-way authentication for the connector. - Alias Attribute that indicates the alias of the key to use when the keystore contains many private keys. If not defined, the first key in the file is used by default. - Key password Key manager password, which is the password for the private key inside the keystore. - Password Store password for the keystore file. This is optional and needed only if the **Key Store Location** is configured. - Algorithm Algorithm used in the keystore. ![Kafka consumer configuration window](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-tls-studio-config.png) ## Configure the Commit Operation 1. Drag the **Commit** operation to the Studio canvas. 2. Configure the **Commit** operation in the **General** tab:

Name

Type

Description

Default Value

Required

Configuration

String

The name of the configuration to use.

x

Consumer commit key

String

The commitKey of the last poll. This operation is valid only when used inside a flow that is using one of the listener sources (Batch message listener or Message listener) which inserts this value as an attribute in the Mule event.

x

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-commit-studio-config-general.png) 3. In the **Advanced** tab, configure the reconnection strategy. ## Configure the Consume Operation 1. Drag the **Consume** operation to the Studio canvas. 2. Configure the **Consume** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Consumption timeout

Number

The number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The unit of time for the timeout property.

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-consume-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Target Variable

String

The name of a variable that stores the operation’s output.

Target Value

String

An expression that evaluates the operation’s output. The outcome of the evaluation is stored in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Publish Operation 1. Drag the **Publish** operation to the Studio canvas. 2. Configure the **Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Topic

String

The topic to publish to.

Partition

Number

(Optional) The topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

![Kafka config panel in General tab for producer config](/mulesoft/current/en-us/kafka-connector/4.10/_images/kafka-publish-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Bulk Publish Operation 1. Drag the **Bulk Publish** operation to the Studio canvas. 2. Configure the **Bulk Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Messages

Array of KafkaMessage

The list of messages to publish.

#[payload]

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

For an example that builds the required `KafkaMessage` array and publishes multiple messages, see [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/4.10/kafka-connector-bulk-publish-example). ## Configure the Seek Operation 1. Drag the **Seek** operation to the Studio canvas. 2. Configure the **Seek** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which to perform the seek operation.

x

Partition

Number

Partition number for which to modify the offset.

x

Offset

Number

Offset value to commit for the configured partition.

x

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure an Input Source Configure an input source for the connector, such as the **Message Consumer** operation:
NameDescription

Configuration

The name of the configuration to use.

Topic

Name of the topic from which to consume messages.

Primary Node Only

Whether to execute this source on only the primary node when running in a cluster.

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

You can also use the **Batch Message Listener** operation as an input source in Apache Kafka Connector:
NameDescription

Connector Configuration

The name of the configuration to use.

Poll timeout

The amount of time to block.

Poll timeout time unit

The time unit for the polling timeout. This combines with poll timeout to define the total timeout for the polling.

Acknowledgment mode

Declares the supported acknowledgment mode type.

Amount of parallel consumers

Declares how many consumers to use in parallel.

Primary Node Only

Whether this source should be executed only on the primary node when running in a cluster.

Streaming Strategy

Define the streaming strategy.

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.11 --- # Using Anypoint Studio to Configure Apache Kafka 4.11 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt In Anypoint Studio, add Anypoint Connector for Apache Kafka (Apache Kafka Connector) to a Mule project, configure the connection to the Kafka cluster, and configure an input source for the connector. To configure a connector in Anypoint Studio: 1. Add the connector to a Mule project. 2. Configure the connector. 3. Configure an input source for the connector. ## Add the Connector in Studio 1. In Studio, create a Mule Project. 2. In the Mule Palette, click **(X) Search in Exchange**. 3. In **Add Modules to Project**, type the name of the connector in the search field. 4. Click the connector name in **Available modules**. 5. Click **Add**. 6. Click **Finish**. ## Add the Connector Using Exchange 1. In Studio, create a Mule Project. 2. Click the Exchange **(X)** icon in the upper-left of the Studio task bar. 3. In Exchange, click **Login** and supply your Anypoint Platform username and password. 4. In Exchange, select **All assets** and search for `Apache Kafka`. 5. Select the connector and click **Add to project**. 6. Follow the prompts to add the connector to the Mule project. ## Consumer Configuration You can create a consumer configuration global element to reference from Apache Kafka Connector. This enables you to apply configuration details to multiple local elements in the flow. 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Consumer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:
Connection TypeField NameDescription

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka consumer clients.

Bean reference

URLs that the consumer can use to connect to the Kafka cluster

Group ID

Default group ID for all the Kafka consumers that use this configuration

Consumer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kafka configuration file (krb5.conf)

File that contains Kerberos configuration information

Consumer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Consumer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Consumer SASL/PLAIN

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Consumer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Producer Configuration Create a producer configuration global element: 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Producer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:

Connection Type

Field Name

Description

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka producer clients.

Bean reference

URLs that the producer can use to connect to the Kafka cluster

Producer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kerberos configuration file (krb5.conf)

File that contains Kerberos configuration information

Producer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Producer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Producer SASL/PLAIN

Username

Username with which to log in to Kafa

Password

Password with which to log in to Kafka

Producer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Configure TLS To enable TLS for your app: 1. Click the **TLS** tab to configure the truststore and keystore: - **Trust Store Configuration** - Path Location of the truststore file. - Password Password for the truststore file. - Type File format of the truststore file. - Algorithm Algorithm the truststore uses. - Insecure Boolean that determines whether or not to validate the truststore. If set to `true`, no validation occurs. The default value is `false`. - **Key Store Configuration** - Type Optionally specify the file format of the keystore file. The default value is `JKS`. - Path Location of the keystore file. This is optional and can be used for two-way authentication for the connector. - Alias Attribute that indicates the alias of the key to use when the keystore contains many private keys. If not defined, the first key in the file is used by default. - Key password Key manager password, which is the password for the private key inside the keystore. - Password Store password for the keystore file. This is optional and needed only if the **Key Store Location** is configured. - Algorithm Algorithm used in the keystore. ![Kafka consumer configuration window](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-tls-studio-config.png) ## Configure the Commit Operation 1. Drag the **Commit** operation to the Studio canvas. 2. Configure the **Commit** operation in the **General** tab:

Name

Type

Description

Default Value

Required

Configuration

String

The name of the configuration to use.

x

Consumer commit key

String

The commitKey of the last poll. This operation is valid only when used inside a flow that is using one of the listener sources (Batch message listener or Message listener) which inserts this value as an attribute in the Mule event.

x

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-commit-studio-config-general.png) 3. In the **Advanced** tab, configure the reconnection strategy. ## Configure the Consume Operation 1. Drag the **Consume** operation to the Studio canvas. 2. Configure the **Consume** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Consumption timeout

Number

The number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The unit of time for the timeout property.

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-consume-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Target Variable

String

The name of a variable that stores the operation’s output.

Target Value

String

An expression that evaluates the operation’s output. The outcome of the evaluation is stored in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Publish Operation 1. Drag the **Publish** operation to the Studio canvas. 2. Configure the **Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Topic

String

The topic to publish to.

Partition

Number

(Optional) The topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

![Kafka config panel in General tab for producer config](/mulesoft/current/en-us/kafka-connector/4.11/_images/kafka-publish-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Bulk Publish Operation 1. Drag the **Bulk Publish** operation to the Studio canvas. 2. Configure the **Bulk Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Messages

Array of KafkaMessage

The list of messages to publish.

#[payload]

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

For an example that builds the required `KafkaMessage` array and publishes multiple messages, see [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/4.11/kafka-connector-bulk-publish-example). ## Configure the Seek Operation 1. Drag the **Seek** operation to the Studio canvas. 2. Configure the **Seek** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which to perform the seek operation.

x

Partition

Number

Partition number for which to modify the offset.

x

Offset

Number

Offset value to commit for the configured partition.

x

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure an Input Source Configure an input source for the connector, such as the **Message Consumer** operation:
NameDescription

Configuration

The name of the configuration to use.

Topic

Name of the topic from which to consume messages.

Primary Node Only

Whether to execute this source on only the primary node when running in a cluster.

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

You can also use the **Batch Message Listener** operation as an input source in Apache Kafka Connector:
NameDescription

Connector Configuration

The name of the configuration to use.

Poll timeout

The amount of time to block.

Poll timeout time unit

The time unit for the polling timeout. This combines with poll timeout to define the total timeout for the polling.

Acknowledgment mode

Declares the supported acknowledgment mode type.

Amount of parallel consumers

Declares how many consumers to use in parallel.

Primary Node Only

Whether this source should be executed only on the primary node when running in a cluster.

Streaming Strategy

Define the streaming strategy.

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.12 --- # Using Anypoint Studio to Configure Apache Kafka 4.12 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt In Anypoint Studio, add Anypoint Connector for Apache Kafka (Apache Kafka Connector) to a Mule project, configure the connection to the Kafka cluster, and configure an input source for the connector. To configure a connector in Anypoint Studio: 1. Add the connector to a Mule project. 2. Configure the connector. 3. Configure an input source for the connector. ## Add the Connector in Studio 1. In Studio, create a Mule Project. 2. In the Mule Palette, click **(X) Search in Exchange**. 3. In **Add Modules to Project**, type the name of the connector in the search field. 4. Click the connector name in **Available modules**. 5. Click **Add**. 6. Click **Finish**. ## Add the Connector Using Exchange 1. In Studio, create a Mule Project. 2. Click the Exchange **(X)** icon in the upper-left of the Studio task bar. 3. In Exchange, click **Login** and supply your Anypoint Platform username and password. 4. In Exchange, select **All assets** and search for `Apache Kafka`. 5. Select the connector and click **Add to project**. 6. Follow the prompts to add the connector to the Mule project. ## Consumer Configuration You can create a consumer configuration global element to reference from Apache Kafka Connector. This enables you to apply configuration details to multiple local elements in the flow. 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Consumer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:
Connection TypeField NameDescription

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka consumer clients.

Bean reference

URLs that the consumer can use to connect to the Kafka cluster

Group ID

Default group ID for all the Kafka consumers that use this configuration

Consumer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kafka configuration file (krb5.conf)

File that contains Kerberos configuration information

Consumer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Consumer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Consumer SASL/PLAIN

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Consumer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Producer Configuration Create a producer configuration global element: 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Producer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:

Connection Type

Field Name

Description

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka producer clients.

Bean reference

URLs that the producer can use to connect to the Kafka cluster

Producer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kerberos configuration file (krb5.conf)

File that contains Kerberos configuration information

Producer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Producer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Producer SASL/PLAIN

Username

Username with which to log in to Kafa

Password

Password with which to log in to Kafka

Producer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Configure TLS To enable TLS for your app: 1. Click the **TLS** tab to configure the truststore and keystore: - **Trust Store Configuration** - Path Location of the truststore file. - Password Password for the truststore file. - Type File format of the truststore file. - Algorithm Algorithm the truststore uses. - Insecure Boolean that determines whether or not to validate the truststore. If set to `true`, no validation occurs. The default value is `false`. - **Key Store Configuration** - Type Optionally specify the file format of the keystore file. The default value is `JKS`. - Path Location of the keystore file. This is optional and can be used for two-way authentication for the connector. - Alias Attribute that indicates the alias of the key to use when the keystore contains many private keys. If not defined, the first key in the file is used by default. - Key password Key manager password, which is the password for the private key inside the keystore. - Password Store password for the keystore file. This is optional and needed only if the **Key Store Location** is configured. - Algorithm Algorithm used in the keystore. ![Kafka consumer configuration window](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-tls-studio-config.png) ## Configure the Commit Operation 1. Drag the **Commit** operation to the Studio canvas. 2. Configure the **Commit** operation in the **General** tab:

Name

Type

Description

Default Value

Required

Configuration

String

The name of the configuration to use.

x

Consumer commit key

String

The commitKey of the last poll. This operation is valid only when used inside a flow that is using one of the listener sources (Batch message listener or Message listener) which inserts this value as an attribute in the Mule event.

x

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-commit-studio-config-general.png) 3. In the **Advanced** tab, configure the reconnection strategy. ## Configure the Consume Operation 1. Drag the **Consume** operation to the Studio canvas. 2. Configure the **Consume** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Consumption timeout

Number

The number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The unit of time for the timeout property.

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-consume-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Target Variable

String

The name of a variable that stores the operation’s output.

Target Value

String

An expression that evaluates the operation’s output. The outcome of the evaluation is stored in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Publish Operation 1. Drag the **Publish** operation to the Studio canvas. 2. Configure the **Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Topic

String

The topic to publish to.

Partition

Number

(Optional) The topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

![Kafka config panel in General tab for producer config](/mulesoft/current/en-us/kafka-connector/4.12/_images/kafka-publish-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Bulk Publish Operation 1. Drag the **Bulk Publish** operation to the Studio canvas. 2. Configure the **Bulk Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Messages

Array of KafkaMessage

The list of messages to publish.

#[payload]

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

For an example that builds the required `KafkaMessage` array and publishes multiple messages, see [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/4.12/kafka-connector-bulk-publish-example). ## Configure the Seek Operation 1. Drag the **Seek** operation to the Studio canvas. 2. Configure the **Seek** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which to perform the seek operation.

x

Partition

Number

Partition number for which to modify the offset.

x

Offset

Number

Offset value to commit for the configured partition.

x

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure an Input Source Configure an input source for the connector, such as the **Message Consumer** operation:
NameDescription

Configuration

The name of the configuration to use.

Topic

Name of the topic from which to consume messages.

Primary Node Only

Whether to execute this source on only the primary node when running in a cluster.

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

You can also use the **Batch Message Listener** operation as an input source in Apache Kafka Connector:
NameDescription

Connector Configuration

The name of the configuration to use.

Poll timeout

The amount of time to block.

Poll timeout time unit

The time unit for the polling timeout. This combines with poll timeout to define the total timeout for the polling.

Acknowledgment mode

Declares the supported acknowledgment mode type.

Amount of parallel consumers

Declares how many consumers to use in parallel.

Primary Node Only

Whether this source should be executed only on the primary node when running in a cluster.

Streaming Strategy

Define the streaming strategy.

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.13 --- # Using Anypoint Studio to Configure Apache Kafka 4.13 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt In Anypoint Studio, add Anypoint Connector for Apache Kafka (Apache Kafka Connector) to a Mule project, configure the connection to the Kafka cluster, and configure an input source for the connector. To configure a connector in Anypoint Studio: 1. Add the connector to a Mule project. 2. Configure the connector. 3. Configure an input source for the connector. ## Add the Connector in Studio 1. In Studio, create a Mule Project. 2. In the Mule Palette, click **(X) Search in Exchange**. 3. In **Add Modules to Project**, type the name of the connector in the search field. 4. Click the connector name in **Available modules**. 5. Click **Add**. 6. Click **Finish**. ## Add the Connector Using Exchange 1. In Studio, create a Mule Project. 2. Click the Exchange **(X)** icon in the upper-left of the Studio task bar. 3. In Exchange, click **Login** and supply your Anypoint Platform username and password. 4. In Exchange, select **All assets** and search for `Apache Kafka`. 5. Select the connector and click **Add to project**. 6. Follow the prompts to add the connector to the Mule project. ## Consumer Configuration You can create a consumer configuration global element to reference from Apache Kafka Connector. This enables you to apply configuration details to multiple local elements in the flow. 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Consumer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:
Connection TypeField NameDescription

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka consumer clients.

Bean reference

URLs that the consumer can use to connect to the Kafka cluster

Group ID

Default group ID for all the Kafka consumers that use this configuration

Consumer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kafka configuration file (krb5.conf)

File that contains Kerberos configuration information

Consumer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Consumer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Consumer SASL/PLAIN

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Consumer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Producer Configuration Create a producer configuration global element: 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Producer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:

Connection Type

Field Name

Description

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka producer clients.

Bean reference

URLs that the producer can use to connect to the Kafka cluster

Producer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kerberos configuration file (krb5.conf)

File that contains Kerberos configuration information

Producer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Producer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Producer SASL/PLAIN

Username

Username with which to log in to Kafa

Password

Password with which to log in to Kafka

Producer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Configure TLS To enable TLS for your app: 1. Click the **TLS** tab to configure the truststore and keystore: - **Trust Store Configuration** - Path Location of the truststore file. - Password Password for the truststore file. - Type File format of the truststore file. - Algorithm Algorithm the truststore uses. - Insecure Boolean that determines whether or not to validate the truststore. If set to `true`, no validation occurs. The default value is `false`. - **Key Store Configuration** - Type Optionally specify the file format of the keystore file. The default value is `JKS`. - Path Location of the keystore file. This is optional and can be used for two-way authentication for the connector. - Alias Attribute that indicates the alias of the key to use when the keystore contains many private keys. If not defined, the first key in the file is used by default. - Key password Key manager password, which is the password for the private key inside the keystore. - Password Store password for the keystore file. This is optional and needed only if the **Key Store Location** is configured. - Algorithm Algorithm used in the keystore. ![Kafka consumer configuration window](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-tls-studio-config.png) ## Configure the Commit Operation 1. Drag the **Commit** operation to the Studio canvas. 2. Configure the **Commit** operation in the **General** tab:

Name

Type

Description

Default Value

Required

Configuration

String

The name of the configuration to use.

x

Consumer commit key

String

The commitKey of the last poll. This operation is valid only when used inside a flow that is using one of the listener sources (Batch message listener or Message listener) which inserts this value as an attribute in the Mule event.

x

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-commit-studio-config-general.png) 3. In the **Advanced** tab, configure the reconnection strategy. ## Configure the Consume Operation 1. Drag the **Consume** operation to the Studio canvas. 2. Configure the **Consume** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Consumption timeout

Number

The number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The unit of time for the timeout property.

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-consume-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Target Variable

String

The name of a variable that stores the operation’s output.

Target Value

String

An expression that evaluates the operation’s output. The outcome of the evaluation is stored in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Publish Operation 1. Drag the **Publish** operation to the Studio canvas. 2. Configure the **Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Topic

String

The topic to publish to.

Partition

Number

(Optional) The topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

![Kafka config panel in General tab for producer config](/mulesoft/current/en-us/kafka-connector/4.13/_images/kafka-publish-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Bulk Publish Operation 1. Drag the **Bulk Publish** operation to the Studio canvas. 2. Configure the **Bulk Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Messages

Array of KafkaMessage

The list of messages to publish.

#[payload]

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

For an example that builds the required `KafkaMessage` array and publishes multiple messages, see [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/4.13/kafka-connector-bulk-publish-example). ## Configure the Seek Operation 1. Drag the **Seek** operation to the Studio canvas. 2. Configure the **Seek** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which to perform the seek operation.

x

Partition

Number

Partition number for which to modify the offset.

x

Offset

Number

Offset value to commit for the configured partition.

x

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure an Input Source Configure an input source for the connector, such as the **Message Consumer** operation:
NameDescription

Configuration

The name of the configuration to use.

Topic

Name of the topic from which to consume messages.

Primary Node Only

Whether to execute this source on only the primary node when running in a cluster.

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

You can also use the **Batch Message Listener** operation as an input source in Apache Kafka Connector:
NameDescription

Connector Configuration

The name of the configuration to use.

Poll timeout

The amount of time to block.

Poll timeout time unit

The time unit for the polling timeout. This combines with poll timeout to define the total timeout for the polling.

Acknowledgment mode

Declares the supported acknowledgment mode type.

Amount of parallel consumers

Declares how many consumers to use in parallel.

Primary Node Only

Whether this source should be executed only on the primary node when running in a cluster.

Streaming Strategy

Define the streaming strategy.

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.14 --- # Using Anypoint Studio to Configure Apache Kafka 4.14 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt In Anypoint Studio, add Anypoint Connector for Apache Kafka (Apache Kafka Connector) to a Mule project, configure the connection to the Kafka cluster, and configure an input source for the connector. To configure a connector in Anypoint Studio: 1. Add the connector to a Mule project. 2. Configure the connector. 3. Configure an input source for the connector. ## Add the Connector in Studio 1. In Studio, create a Mule Project. 2. In the Mule Palette, click **(X) Search in Exchange**. 3. In **Add Modules to Project**, type the name of the connector in the search field. 4. Click the connector name in **Available modules**. 5. Click **Add**. 6. Click **Finish**. ## Add the Connector Using Exchange 1. In Studio, create a Mule Project. 2. Click the Exchange **(X)** icon in the upper-left of the Studio task bar. 3. In Exchange, click **Login** and supply your Anypoint Platform username and password. 4. In Exchange, select **All assets** and search for `Apache Kafka`. 5. Select the connector and click **Add to project**. 6. Follow the prompts to add the connector to the Mule project. ## Consumer Configuration You can create a consumer configuration global element to reference from Apache Kafka Connector. This enables you to apply configuration details to multiple local elements in the flow. 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Consumer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:
Connection TypeField NameDescription

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka consumer clients.

Bean reference

URLs that the consumer can use to connect to the Kafka cluster

Group ID

Default group ID for all the Kafka consumers that use this configuration

Consumer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kafka configuration file (krb5.conf)

File that contains Kerberos configuration information

Consumer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Consumer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Consumer SASL/PLAIN

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Consumer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Producer Configuration Create a producer configuration global element: 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Producer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:

Connection Type

Field Name

Description

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka producer clients.

Bean reference

URLs that the producer can use to connect to the Kafka cluster

Producer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kerberos configuration file (krb5.conf)

File that contains Kerberos configuration information

Producer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Producer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Producer SASL/PLAIN

Username

Username with which to log in to Kafa

Password

Password with which to log in to Kafka

Producer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Configure TLS To enable TLS for your app: 1. Click the **TLS** tab to configure the truststore and keystore: - **Trust Store Configuration** - Path Location of the truststore file. - Password Password for the truststore file. - Type File format of the truststore file. - Algorithm Algorithm the truststore uses. - Insecure Boolean that determines whether or not to validate the truststore. If set to `true`, no validation occurs. The default value is `false`. - **Key Store Configuration** - Type Optionally specify the file format of the keystore file. The default value is `JKS`. - Path Location of the keystore file. This is optional and can be used for two-way authentication for the connector. - Alias Attribute that indicates the alias of the key to use when the keystore contains many private keys. If not defined, the first key in the file is used by default. - Key password Key manager password, which is the password for the private key inside the keystore. - Password Store password for the keystore file. This is optional and needed only if the **Key Store Location** is configured. - Algorithm Algorithm used in the keystore. ![Kafka consumer configuration window](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-tls-studio-config.png) ## Configure the Commit Operation 1. Drag the **Commit** operation to the Studio canvas. 2. Configure the **Commit** operation in the **General** tab:

Name

Type

Description

Default Value

Required

Configuration

String

The name of the configuration to use.

x

Consumer commit key

String

The commitKey of the last poll. This operation is valid only when used inside a flow that is using one of the listener sources (Batch message listener or Message listener) which inserts this value as an attribute in the Mule event.

x

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-commit-studio-config-general.png) 3. In the **Advanced** tab, configure the reconnection strategy. ## Configure the Consume Operation 1. Drag the **Consume** operation to the Studio canvas. 2. Configure the **Consume** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Consumption timeout

Number

The number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The unit of time for the timeout property.

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-consume-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Target Variable

String

The name of a variable that stores the operation’s output.

Target Value

String

An expression that evaluates the operation’s output. The outcome of the evaluation is stored in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Publish Operation 1. Drag the **Publish** operation to the Studio canvas. 2. Configure the **Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Topic

String

The topic to publish to.

Partition

Number

(Optional) The topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

![Kafka config panel in General tab for producer config](/mulesoft/current/en-us/kafka-connector/4.14/_images/kafka-publish-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Bulk Publish Operation 1. Drag the **Bulk Publish** operation to the Studio canvas. 2. Configure the **Bulk Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Messages

Array of KafkaMessage

The list of messages to publish.

#[payload]

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

For an example that builds the required `KafkaMessage` array and publishes multiple messages, see [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/4.14/kafka-connector-bulk-publish-example). ## Configure the Seek Operation 1. Drag the **Seek** operation to the Studio canvas. 2. Configure the **Seek** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which to perform the seek operation.

x

Partition

Number

Partition number for which to modify the offset.

x

Offset

Number

Offset value to commit for the configured partition.

x

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure an Input Source Configure an input source for the connector, such as the **Message Consumer** operation:
NameDescription

Configuration

The name of the configuration to use.

Topic

Name of the topic from which to consume messages.

Primary Node Only

Whether to execute this source on only the primary node when running in a cluster.

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

You can also use the **Batch Message Listener** operation as an input source in Apache Kafka Connector:
NameDescription

Connector Configuration

The name of the configuration to use.

Poll timeout

The amount of time to block.

Poll timeout time unit

The time unit for the polling timeout. This combines with poll timeout to define the total timeout for the polling.

Acknowledgment mode

Declares the supported acknowledgment mode type.

Amount of parallel consumers

Declares how many consumers to use in parallel.

Primary Node Only

Whether this source should be executed only on the primary node when running in a cluster.

Streaming Strategy

Define the streaming strategy.

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Apache Kafka Connector version: 4.15 is-latest-version: true --- # Using Anypoint Studio to Configure Apache Kafka 4.15 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt In Anypoint Studio, add Anypoint Connector for Apache Kafka (Apache Kafka Connector) to a Mule project, configure the connection to the Kafka cluster, and configure an input source for the connector. To configure a connector in Anypoint Studio: 1. Add the connector to a Mule project. 2. Configure the connector. 3. Configure an input source for the connector. ## Add the Connector in Studio 1. In Studio, create a Mule Project. 2. In the Mule Palette, click **(X) Search in Exchange**. 3. In **Add Modules to Project**, type the name of the connector in the search field. 4. Click the connector name in **Available modules**. 5. Click **Add**. 6. Click **Finish**. ## Add the Connector Using Exchange 1. In Studio, create a Mule Project. 2. Click the Exchange **(X)** icon in the upper-left of the Studio task bar. 3. In Exchange, click **Login** and supply your Anypoint Platform username and password. 4. In Exchange, select **All assets** and search for `Apache Kafka`. 5. Select the connector and click **Add to project**. 6. Follow the prompts to add the connector to the Mule project. ## Consumer Configuration You can create a consumer configuration global element to reference from Apache Kafka Connector. This enables you to apply configuration details to multiple local elements in the flow. 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Consumer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:
Connection TypeField NameDescription

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka consumer clients.

Bean reference

URLs that the consumer can use to connect to the Kafka cluster

Group ID

Default group ID for all the Kafka consumers that use this configuration

Consumer SASL/AWS IAM

AWS Region

AWS Region where the Amazon MSK cluster is located

Consumer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kafka configuration file (krb5.conf)

File that contains Kerberos configuration information

Consumer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Consumer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Consumer SASL/PLAIN

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Consumer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Producer Configuration Create a producer configuration global element: 1. On the Studio canvas, click **Global Elements**. 2. Click **Create** and expand **Connector Configuration**. 3. Select **Apache Kafka Producer configuration** and click **OK**. 4. In the **Connection** field, select a connection type. 5. Enter values for the following fields:

Connection Type

Field Name

Description

All connection types

Bootstrap Server URLs

List of host-port pairs that establish the initial connection to the Kafka cluster. Set this value to the bootstrap.servers value you must provide to Kafka producer clients.

Bean reference

URLs that the producer can use to connect to the Kafka cluster

Producer SASL/AWS IAM

AWS Region

AWS Region where the Amazon MSK cluster is located

Producer Kerberos

Principal

Entity for the Apache Kafka system to authenticate

Service name

Kerberos principal name associated with Kafka Connector

Kerberos configuration file (krb5.conf)

File that contains Kerberos configuration information

Producer SASL/OAUTHBEARER

Client ID

Unique identifier assigned to the OAuth client, facilitating secure authentication and authorization processes

Client Secret

Confidential authentication credential assigned to the OAuth client

OAuth Token Endpoint

URL where clients can request access tokens from the authentication server

Producer SASL/SCRAM

Username

Username with which to log in to Kafka

Password

Password with which to log in to Kafka

Encryption type

Encryption algorithm used by SCRAM

Producer SASL/PLAIN

Username

Username with which to log in to Kafa

Password

Password with which to log in to Kafka

Producer SASL/TOKEN

Token ID

ID of the token

Token HMAC

Token HMAC

Encryption type

Encryption algorithm used by SCRAM

## Configure TLS To enable TLS for your app: 1. Click the **TLS** tab to configure the truststore and keystore: - **Trust Store Configuration** - Path Location of the truststore file. - Password Password for the truststore file. - Type File format of the truststore file. - Algorithm Algorithm the truststore uses. - Insecure Boolean that determines whether or not to validate the truststore. If set to `true`, no validation occurs. The default value is `false`. - **Key Store Configuration** - Type Optionally specify the file format of the keystore file. The default value is `JKS`. - Path Location of the keystore file. This is optional and can be used for two-way authentication for the connector. - Alias Attribute that indicates the alias of the key to use when the keystore contains many private keys. If not defined, the first key in the file is used by default. - Key password Key manager password, which is the password for the private key inside the keystore. - Password Store password for the keystore file. This is optional and needed only if the **Key Store Location** is configured. - Algorithm Algorithm used in the keystore. ![Kafka consumer configuration window](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-tls-studio-config.png) ## Configure the Commit Operation 1. Drag the **Commit** operation to the Studio canvas. 2. Configure the **Commit** operation in the **General** tab:

Name

Type

Description

Default Value

Required

Configuration

String

The name of the configuration to use.

x

Consumer commit key

String

The commitKey of the last poll. This operation is valid only when used inside a flow that is using one of the listener sources (Batch message listener or Message listener) which inserts this value as an attribute in the Mule event.

x

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-commit-studio-config-general.png) 3. In the **Advanced** tab, configure the reconnection strategy. ## Configure the Consume Operation 1. Drag the **Consume** operation to the Studio canvas. 2. Configure the **Consume** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Consumption timeout

Number

The number of time units that this operation waits for receiving messages.

Timeout time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The unit of time for the timeout property.

![Kafka config panel in General tab for consumer config](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-consume-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Target Variable

String

The name of a variable that stores the operation’s output.

Target Value

String

An expression that evaluates the operation’s output. The outcome of the evaluation is stored in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Publish Operation 1. Drag the **Publish** operation to the Studio canvas. 2. Configure the **Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Topic

String

The topic to publish to.

Partition

Number

(Optional) The topic partition.

Key

Binary

(Optional) Key for the published message.

Message

Binary

(Optional) Message content of the message.

#[payload]

Headers

Object

(Optional) Headers for the message.

![Kafka config panel in General tab for producer config](/mulesoft/current/en-us/kafka-connector/latest/_images/kafka-publish-studio-config.png) 3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure the Bulk Publish Operation 1. Drag the **Bulk Publish** operation to the Studio canvas. 2. Configure the **Bulk Publish** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Messages

Array of KafkaMessage

The list of messages to publish.

#[payload]

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Transactional Action

Enumeration, one of:

  • ALWAYS_JOIN

  • JOIN_IF_POSSIBLE

  • NOT_SUPPORTED

The type of joining action that operations can take regarding transactions.

JOIN_IF_POSSIBLE

Target Variable

String

The name of a variable to store the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

For an example that builds the required `KafkaMessage` array and publishes multiple messages, see [Publish Multiple Messages with Bulk Publish](https://docs.mulesoft.com/kafka-connector/latest/kafka-connector-bulk-publish-example). ## Configure the Seek Operation 1. Drag the **Seek** operation to the Studio canvas. 2. Configure the **Seek** operation in the **General** tab:
NameTypeDescriptionDefault ValueRequired

Configuration

String

Name of the configuration to use.

x

Topic

String

Name of the topic on which to perform the seek operation.

x

Partition

Number

Partition number for which to modify the offset.

x

Offset

Number

Offset value to commit for the configured partition.

x

3. Configure the following settings in the **Advanced** tab:
NameTypeDescriptionDefault ValueRequired

Operation Timeout

Number

Operation Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Reconnection Strategy

  • reconnect

  • reconnect-forever

A retry strategy in case of connectivity errors.

## Configure an Input Source Configure an input source for the connector, such as the **Message Consumer** operation:
NameDescription

Configuration

The name of the configuration to use.

Topic

Name of the topic from which to consume messages.

Primary Node Only

Whether to execute this source on only the primary node when running in a cluster.

Streaming Strategy

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Configure to use repeatable streams.

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

You can also use the **Batch Message Listener** operation as an input source in Apache Kafka Connector:
NameDescription

Connector Configuration

The name of the configuration to use.

Poll timeout

The amount of time to block.

Poll timeout time unit

The time unit for the polling timeout. This combines with poll timeout to define the total timeout for the polling.

Acknowledgment mode

Declares the supported acknowledgment mode type.

Amount of parallel consumers

Declares how many consumers to use in parallel.

Primary Node Only

Whether this source should be executed only on the primary node when running in a cluster.

Streaming Strategy

Define the streaming strategy.

  • repeatable-in-memory-stream

  • repeatable-file-store-stream

  • non-repeatable-stream

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Reconnection Strategy

A retry strategy in case of connectivity errors.

  • reconnect

  • reconnect-forever

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com)