# mulesoft Amazon Kinesis Connector - Full Documentation --- product: Amazon Kinesis Connector version: 1.0 --- # Amazon Kinesis Data Streams Connector 1.0 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Amazon Kinesis Data Streams (Amazon Kinesis Data Streams Connector) provides access to standard Amazon Kinesis Data Stream operations using Anypoint Platform. Amazon Kinesis is a streaming solution built by Amazon AWS to collect and process large streams of data records in real time. To learn more, see the Amazon Kinesis official documentation. For compatibility information and fixed issues, see the Amazon Kinesis Data Streams Release Notes. > [!NOTE] Amazon Kinesis Data Streams Connector doesn’t support Windows because it uses the Kinesis > Producer Library (KPL), which deprecated Windows support starting in version 0.14.0. ## Before You Begin To use this connector, you must be familiar with: - The Amazon Kinesis Data Streams API - Anypoint Connectors - Mule runtime engine (Mule) - Elements and global elements in a Mule flow - Creating a Mule app using Anypoint Studio or Anypoint Code Builder Before creating an app, you must have: - AWS Identity and Access Management (IAM) Access Key credentials with access rights to the Kinesis, DynamoDB, and CloudWatch services. You can find the required permissions [here](https://docs.aws.amazon.com/streams/latest/dev/tutorial-stock-data-kplkcl-iam.html). - Access to the Amazon Kinesis target resource ## Common Use Cases for the Connector Common use cases for Kinesis Data Streams connector include the following: - Troubleshooting Collect log and event data from sources such as servers, desktops, and mobile devices. Then continuously process the data, generate metrics, power live dashboards, and put the aggregated data into data stores such as Amazon S3. - Real-time analytics Run real-time analytics on high-frequency event data such as sensor data or social media data collected by Amazon Kinesis Data Streams. These analytics enable you to gain insights from your data at a frequency of minutes instead of a frequency of hours or days. - Mobile data capture Have your mobile applications push data to Amazon Kinesis data streams from hundreds of thousands of devices, making the data available to you as soon as it is produced on the mobile devices. For examples, see [Examples](https://docs.mulesoft.com/amazon-kinesis-connector/1.0/amazon-kinesis-connector-examples). ## Audience - New users, read: - [Using Anypoint Studio to Configure the Connector](https://docs.mulesoft.com/amazon-kinesis-connector/1.0/amazon-kinesis-connector-studio) to create your Mule app - [Examples](https://docs.mulesoft.com/amazon-kinesis-connector/1.0/amazon-kinesis-connector-examples), which provide one or more use cases for the connector - Advanced users, read: - [XML and Maven Support](https://docs.mulesoft.com/amazon-kinesis-connector/1.0/amazon-kinesis-connector-xml-maven) - [Examples](https://docs.mulesoft.com/amazon-kinesis-connector/1.0/amazon-kinesis-connector-examples), which provides one or more use cases for the connector ## Authentication Types Amazon Kinesis Data Streams Connector supports Access Key authentication. With this type of authentication, the connector authenticates to AWS by using an access key, secret key, and optionally, a role. For information about configuring authentication, see [Using Anypoint Studio to Configure Amazon Kinesis Data Streams](https://docs.mulesoft.com/amazon-kinesis-connector/1.0/amazon-kinesis-connector-studio). ## Using Exchange Templates and Examples [Anypoint Exchange](https://www.mulesoft.com/exchange/) provides templates that you can use as starting points for your apps and examples that illustrate a complete solution. ## Next Step After you complete the prerequisites and try the templates and examples, you are ready to create your own app and configure the connector using [Anypoint Studio](https://docs.mulesoft.com/amazon-kinesis-connector/1.0/amazon-kinesis-connector-studio). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Use Exchange to Discover Connectors, Templates, and Examples](https://docs.mulesoft.com/connectors/introduction/intro-use-exchange) - [Salesforce Help](https://help.salesforce.com) --- product: Amazon Kinesis Connector version: 1.0 --- # Amazon Kinesis Data Streams Connector 1.0 Examples > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to use Anypoint Connector for Amazon Kinesis Data Streams (Amazon Kinesis Data Streams Connector) to put data records and listen from an Amazon Kinesis data stream. > [!NOTE] Amazon Kinesis Data Streams Connector supports > [Enhanced Fan-Out (EFO)](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html) > by default. This means that the connector automatically subscribes to all shards of the configured > stream using EFO. > > The connector’s behavior aligns with AWS Kinesis Client Library (KCL) version 2.0 and later. The > connector uses the same implementation as shown in > [Develop enhanced fan-out consumers using KCL 2.x in Java](https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-kcl-java.html). Before you try the example, access Anypoint Studio (Studio) and verify that the **Mule Palette** view displays entries for Kinesis. If not, follow the instructions in [Add the Connector to Your Project](https://docs.mulesoft.com/amazon-kinesis-connector/1.0/amazon-kinesis-connector-studio#add-connector-to-project). ## Flows for This Example The following screenshots show the Anypoint Studio app flows for this example: - This flow uses the **Put Records** operation to put data into the Amazon Kinesis data stream. In this flow, **HTTP > Listener** receives a query parameter named **data** that sets the payload for the **Put Record** operation. It uses a query parameter named **partitionKey** to set the partition key value for the **Put Record** operation. ![Put Record operation flow](/mulesoft/current/en-us/amazon-kinesis-connector/1.0/_images/amazon-kinesis-example-put-record.png) - This flow uses the **Listener** source to listen for new data records: ![Listener source flow](/mulesoft/current/en-us/amazon-kinesis-connector/1.0/_images/amazon-kinesis-example-listener.png) - This flow uses the **Listener** source to listen for new data records. After receiving the new records, it calls the **Checkpoint** operation to set a manual checkpoint: ![Checkpoint operation flow](/mulesoft/current/en-us/amazon-kinesis-connector/1.0/_images/amazon-kinesis-example-checkpoint.png) ## XML Code for This Example ```xml ``` ### See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Amazon Kinesis Connector version: 1.0 --- # Amazon Kinesis Data Streams Connector 1.0 Reference > For the full documentation index, see: https://docs.mulesoft.com/llms.txt > [!NOTE] Amazon Kinesis Data Streams Connector doesn’t support Windows because it uses the Kinesis > Producer Library (KPL), which deprecated Windows support starting in version 0.14.0. ## Configurations --- ### Config The configuration for the connector. You can use this configuration for both the producer and consumer. #### Parameters
NameTypeDescriptionDefault ValueRequired

Name

String

The name for this configuration. Connectors reference the configuration with this name.

x

Connection

The connection types to provide to this configuration.

x

Aggregation Enabled

Boolean

When set to true, this option packs multiple user data records into a single Amazon Kinesis data record.

true

Response Timeout

Number

The maximum total time that can elapse between the start of an HTTP request and when the responses are received. If this time is exceeded, the request times out.

30

Request Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

A time unit that qualifies the Response Timeout value

SECONDS

Record Max Buffered Time

Number

Maximum amount of time that an Amazon Kinesis data record spends buffering before it is sent. Records can be sent earlier than this value, depending on other buffering limits.

3

Record Max Buffered Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

A time unit that qualifies the Record Max Buffered Time value

SECONDS

Aggregation Max Count

Number

Maximum number of items to pack into an aggregated Amazon Kinesis data record.

4294967295

Aggregation Max Size

Number

Maximum data size to pack into an aggregated Amazon Kinesis data record.

51200

Aggregation Max Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

A data unit that qualifies the Aggregation Max Size value

BYTE

Collection Max Count

Number

Maximum number of items to pack into a batch request.

500

Collection Max Size

Number

Maximum amount of data to send with a Put Record operation.

5242880

Collection Max Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

A data unit that qualifies the Collection Max Size value

BYTE

Connection Timeout

Number

The amount of time to wait when initially establishing the TCP connection between the connector and Amazon Kinesis server before throwing an exception if the connection fails

30

Connection Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The time unit for the Connection Timeout value

SECONDS

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput.

24

Min Connections

Number

Minimum number of connections to keep open to the backend. In most cases, there is no need to increase value.

1

Rate Limit

Number

Limits the number of data records per second and number of bytes per second sent to a shard. This limit can reduce bandwidth and CPU cycle wastage from sending requests that will fail from throttling. The default value, 150%, enables a single producer instance to completely saturate the allowance for a shard. To reduce throttling errors rather than completely saturate the shard, consider reducing this setting.

150%

Record TTL

Number

Time limit for data records to be put. Records that are not successfully put within this limit fail. When setting this value, take into consideration the fact that data records still incur network latency after they leave the Amazon Kinesis Producer Library (KPL).

30

Record TTL Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The time unit for the Record TTL value.

SECONDS

Thread Pool Size

Number

Maximum number of threads with which to configure the native process thread pool. The default value, 0, enables the KPL process to choose the size of the thread pool.

0

#### Connection Types ##### Kinesis Connection Connects to AWS Kinesis through the use of a user-specified access and secret key and, optionally, a role. These credentials securely sign requests sent to AWS services. ###### Parameters
NameTypeDescriptionDefault ValueRequired

Proxy Configuration

Proxy connection settings for outbound connections. This setting applies to the Listener source and Checkpoint operation. It does not affect the Put Record operation.

TLS Configuration

Protocol to use for communication. When using the HTTPS protocol, the HTTP communication is secured using TLS or SSL. If HTTPS is configured as the protocol, then, at a minimum, you must configure the keystore in the tls:context child element of the listener-config.

This setting applies to the Listener source and Checkpoint operation. It does not affect the Put Record operation.

HTTP

Access Key

String

Access key provided by Amazon.

x

Secret Key

String

Secret key provided by Amazon.

x

Region Endpoint

String

Topic region endpoint

us-east-1

Role

Role configuration

Connection Timeout

Number

The amount of time to wait when initially establishing the TCP connection between the connector and Amazon Kinesis server before throwing an exception if the connection fails

30

Connection Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The time unit for the Connection Timeout field.

SECONDS

Response Timeout

Number

The maximum time to wait between the time an HTTP request is sent and when a response is received

30

Response Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Response Timeout value.

SECONDS

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.

## Supported Operations - [Checkpoint](#Checkpoint) - [Put Record](#PutRecord) ## Associated Sources - [Listener](#Listener) ## Operations ## Checkpoint `` The operation checkpoint position of a listener into the Amazon DynamoDB table. Upon failover, the connector starts fetching records that are located after the check pointed position. The **Checkpoint** operation does not modify the current position of the listener. ### Parameters
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Stream Name

String

The stream name.

x

Application Name

String

Name of the DynamoDB table that holds data about the current stream positions.

x

Shard ID

String

The shard identifier.

Sequence Number

String

A sequence number at which to checkpoint in the shard.

Subsequence Number

Number

A subsequence number at which to checkpoint within the shard.

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component

x

### For Configurations - [Config](#Config) ### Throws - KINESIS:ILLEGAL\_STATE - KINESIS:INVALID\_ARGUMENT - KINESIS:PROVISIONED\_THROUGHPUT\_EXCEEDED - KINESIS:TIMEOUT - KINESIS:UNAUTHORIZED ## Put Record `` Puts a data record into an Amazon Kinesis data stream. ### Parameters
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Stream Name

String

The stream name

x

Partition Key

String

The partition key. If not provided, the connector generates a UUID for this value.

Explicit Hash Key

String

Overrides the Amazon Kinesis Data Streams explicitHashKey value, which is normally computed with using MD5 function of the data record partition key.

Data

Any

The content to put in the Amazon Kinesis data stream. This content can be up to 1 MB in size.

#[payload]

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component

x

Streaming Strategy

Disables the repeatable stream functionality and uses non-repeatable streams to have lower performance overhead, memory use, and costs

Target Variable

String

Name of the variable in which to store the operation’s output

Target Value

String

Expression that evaluates the operation’s output. The expression outcome is stored in the target variable.

#[payload]

Reconnection Strategy

A retry strategy in case of connectivity errors

### Output

Type

Any

### For Configurations - [Config](#Config) ### Throws - KINESIS:CONNECTIVITY - KINESIS:INVALID\_ARGUMENT - KINESIS:PROCESSING - KINESIS:RETRY\_EXHAUSTED - KINESIS:TIMEOUT - KINESIS:UNAUTHORIZED ## Sources ## Listener `` Uses the Amazon Kinesis Client Library (KCL) to listen from a specified point in a stream. > [!NOTE] Due to varying batch sizes, checkpoints can occur in the incorrect sequence, potentially > leading to transaction loss in certain cases. Set `maxConcurrency=1` to ensure transactions are > processed sequentially. ### Parameters
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Output Mime Type

String

The MIME type of the payload that this operation outputs.

Output Encoding

String

The encoding of the payload that this operation outputs.

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component

x

Primary Node Only

Boolean

Whether this listener executes on the primary node only when running in a cluster

On Capacity Overload

Enumeration, one of:

  • WAIT

  • DROP

Strategy that Mule applies when the flow receives more messages than it can handle

WAIT

Redelivery Policy

Policy for processing the redelivery of a message

Stream Name

String

The stream name.

x

Application Name

String

Name of the DynamoDB table that holds data about current stream positions. If the table doesn’t exist, then the connector creates a new one at the initial position.

x

Shard Configs

Array of Shard Config

The shard configuration. If specified, the listener listens only from selected shards.

Max Batch Size

Number

Maximum number of data records a batch can carry. A batch is the number of records that the listener receives from the stream.

10000

Max Leases For Worker

Number

Number of shards to which the listener can subscribe in parallel.

2137483647

Shard Prioritization

Enumeration, one of:

  • PARENTS_FIRST_SHARD_PRIORITIZATION

  • NO_OP_SHARD_PRIORITIZATION

Logic used to prioritize or filter the shards before their execution.

PARENTS_FIRST_SHARD_PRIORITIZATION Prioritizes parent shards first and limits the number of shards that are available for initialization based on their depth (Max Depth field).

NO_OP_SHARD_PRIORITIZATION Returns the original list of shards without any modifications.

NO_OP_SHARD_PRIORITIZATION

Max Depth

Number

Used by the PARENTS_FIRST_SHARD_PRIORITIZATION shard prioritization option. Any shard that is deeper than this value is excluded from processing.

Read Capacity Units

Number

The maximum number of strongly consistent reads consumed per second before Amazon DynamoDB returns a ThrottlingException.

10

Write Capacity Units

Number

The maximum number of writes consumed per second before Amazon DynamoDB returns a ThrottlingException.

10

Failover Time

Number

Duration of a lease for a worker. This parameter also determines the frequency with which a worker looks for new leases to work on. The frequency is roughly twice the specified value.

10

Failover Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Failover Time field.

SECONDS

Checkpoint On Complete

Boolean

If true, the listener checkpoints the current reading position to Amazon DynamoDB after data records are processed by the owning flow, without taking into account whether the processing finished successfully.

true

Absolute Position

Enumeration, one of:

  • LATEST

  • FIRST

Whether the listener listens from the latest data record or the oldest data record

Timestamp

DateTime

Date and time from which the listener listens

Reconnection Strategy

A retry strategy in case of connectivity errors

### Output

Type

Array of binary messages with [Record Attributes]

### For Configurations - [Config](#Config) ## Types ### Proxy Configuration
FieldTypeDescriptionDefault ValueRequired

Host

String

Host where the proxy requests are sent.

x

Port

Number

Port where the proxy requests are sent.

x

Username

String

The username to authenticate against the proxy.

Password

String

The password to authenticate against the proxy.

Non Proxy Hosts

Array of String

A list of hosts against which the proxy should not be used.

Ntlm Domain

String

The domain to authenticate against the proxy.

### TLS
FieldTypeDescriptionDefault ValueRequired

Enabled Protocols

String

A comma-separated list of protocols enabled for this context.

Enabled Cipher Suites

String

A comma-separated list of cipher suites enabled for this context.

Trust Store

Key Store

Revocation Check

### Trust Store
FieldTypeDescriptionDefault ValueRequired

Path

String

The location (which is resolved relative to the current classpath and file system, if possible) of the truststore.

Password

String

The password used to protect the truststore.

Type

String

The type of store used.

Algorithm

String

The algorithm used by the truststore.

Insecure

Boolean

If set to true, no certificate validations are performed, which makes connections vulnerable to attacks. Use at your own risk.

### Key Store
FieldTypeDescriptionDefault ValueRequired

Path

String

The location (which is resolved relative to the current classpath and file system, if possible) of the keystore.

Type

String

The type of store used.

Alias

String

When the keystore contains many private keys, this attribute indicates the alias of the key that should be used. If not defined, the first key in the file is used by default.

Key Password

String

The password used to protect the private key.

Password

String

The password used to protect the keystore.

Algorithm

String

The algorithm used by the keystore.

### Standard Revocation Check
FieldTypeDescriptionDefault ValueRequired

Only End Entities

Boolean

Verifies the last element of the certificate chain only.

Prefer Crls

Boolean

Tries CRL instead of OCSP first.

No Fallback

Boolean

Does not use the secondary checking method.

Soft Fail

Boolean

Avoids a certification failure when the revocation server cannot be reached or is busy.

### Custom Ocsp Responder
FieldTypeDescriptionDefault ValueRequired

Url

String

The URL of the OCSP responder.

Cert Alias

String

Alias of the signing certificate for the OCSP response (must be in the truststore), if present.

### Crl File
FieldTypeDescriptionDefault ValueRequired

Path

String

The path to the CRL file.

### Role
FieldTypeDescriptionDefault ValueRequired

Arn

String

The Amazon Resource Name (ARN) of the role to assume.

x

External Id

String

A unique identifier that might be required when you assume a role in another account. If the administrator of the account to which the role belongs provides an external ID, then provide that value in this field.

Duration

Number

The duration of the role session.

3600

Duration Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Duration field

SECONDS

Referred Policy Arns

Array of String

The Amazon Resource Names (ARNs) of the IAM-managed policies to use as managed session policies. The policies must exist in the same account as the role.

Tags

Object

A list of session tags that you want to pass. Each session tag consists of a key name and an associated value.

### Reconnection
FieldTypeDescriptionDefault ValueRequired

Fails Deployment

Boolean

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.

Reconnection Strategy

The reconnection strategy to use.

### Reconnect
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often in milliseconds to reconnect

Count

Number

How many reconnection attempts to make.

### Reconnect Forever
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often in milliseconds to reconnect

### Record Attributes
FieldTypeDescriptionDefault ValueRequired

Aggregated

Boolean

Approximate Arrival Timestamp

String

Encryption Type

String

Partition Key

String

Partition Key

String

Sequence Number

String

Shard Id

String

Sub Sequence Number

Number

### Redelivery Policy
FieldTypeDescriptionDefault ValueRequired

Max Redelivery Count

Number

The maximum number of times a message can be redelivered and processed unsuccessfully before triggering a process-failed message

Message Digest Algorithm

String

The secure hashing algorithm to use.

SHA-256

Message Identifier

The strategy used to identify the messages.

Object Store

ObjectStore

The object store that stores the redelivery counter for each message.

### Redelivery Policy Message Identifier
FieldTypeDescriptionDefault ValueRequired

Use Secure Hash

Boolean

Whether to use a secure hash algorithm to identify a redelivered message.

Id Expression

String

One or more expressions to use to determine when a message was redelivered. This property can be set only if the value of the Use Secure Hash field is false.

### Shard Config
FieldTypeDescriptionDefault ValueRequired

Shard Id

String

The shard identifier.

x

Shard Initial Position

The shard initial position.

x

### Shard Initial Position
FieldTypeDescriptionDefault ValueRequired

At Sequence Number

String

The listener starts listening at this position in the data stream.

After Sequence Number

String

The listener starts listening after this position in the data stream.

Absolute Position

Enumeration, one of:

  • LATEST

  • FIRST

The listener starts listening from the latest or oldest record, depending on the value.

Timestamp

DateTime

### Repeatable In Memory Stream
FieldTypeDescriptionDefault ValueRequired

Initial Buffer Size

Number

The amount of memory to allocate to consume the stream and provide random access to it. If the stream contains more data than can fit into this buffer, then the buffer expands according to the Buffer Size Increment value, with an upper limit of the Max In Memory Size value.

Buffer Size Increment

Number

This is by how much the buffer size expands if it exceeds its initial size. Setting a value of zero or lower means that the buffer should not expand, meaning that a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised when the buffer is full.

Max Buffer Size

Number

The maximum amount of memory to use. If more memory is used, then a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value less than or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit in which the fields in this table are expressed

### Repeatable File Store Stream
FieldTypeDescriptionDefault ValueRequired

In Memory Size

Number

Maximum memory that the stream should use to keep data in memory. If more than that is consumed, content on the disk is buffered.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit in which the In Memory Size value is expressed

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Reference](#) - [Salesforce Help](https://help.salesforce.com) --- product: Amazon Kinesis Connector version: 1.0 --- # Amazon Kinesis Data Streams Connector 1.0 XML and Maven Support > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Although you can manually code a Mule app in XML, it is more efficient to use Anypoint Studio. If you manually code a Mule runtime engine (Mule) app in XML either from the Anypoint Studio XML editor or from a text editor, you can access the connector from your app by adding reference statements to both your XML Mule flow and the Apache Maven `pom.xml` file. When you add the connector in Studio, Studio automatically populates the XML code with the connector’s namespace and schema location and it also adds a dependency to the `pom.xml` file. ## Add a Namespace for the Connector Paste the following code inside the `mule` tag of the header of your configuration XML: ```xml http://www.mulesoft.org/schema/mule/amazon-kinesis http://www.mulesoft.org/schema/mule/amazon-kinesis/current/mule-amazon-kinesis.xsd ``` This example shows how the namespace and schema statements are placed in the XML: ```xml com.mulesoft.connectors amazon-kinesis-data-streams-connector x.x.x mule-plugin ``` Replace `x.x.x` with the version that corresponds to the connector you are using. To obtain the most up-to-date `pom.xml` file information: 1. Go to [Anypoint Exchange](https://www.mulesoft.com/exchange/). 2. In Exchange, click **Login** and supply your Anypoint Platform username and password. 3. In Exchange, search for `amazon kinesis`. 4. Select the connector. 5. Click **Dependency Snippets** near the upper right of the screen. ## Next Step After completing your namespace and `pom.xml` file, you can try the [Examples](https://docs.mulesoft.com/amazon-kinesis-connector/1.0/amazon-kinesis-connector-examples), which provides one or more use cases for the connector. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Amazon Kinesis Connector version: 2.0 is-latest-version: true --- # Amazon Kinesis Data Streams Connector 2.0 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Connector for Amazon Kinesis Data Streams (Amazon Kinesis Data Streams Connector) provides access to standard Amazon Kinesis Data Stream operations using Anypoint Platform. Amazon Kinesis is a streaming solution built by Amazon AWS to collect and process large streams of data records in real time. To learn more, see the Amazon Kinesis official documentation. For compatibility information and fixed issues, see the Amazon Kinesis Data Streams Release Notes. > [!NOTE] Amazon Kinesis Data Streams Connector doesn’t support Windows because it uses the Kinesis > Producer Library (KPL), which deprecated Windows support starting in version 0.14.0. ## Before You Begin To use this connector, you must be familiar with: - The Amazon Kinesis Data Streams API - Anypoint Connectors - Mule runtime engine (Mule) - Elements and global elements in a Mule flow - Creating a Mule app using Anypoint Studio or Anypoint Code Builder Before creating an app, you must have: - AWS Identity and Access Management (IAM) Access Key credentials with access rights to the Kinesis, DynamoDB, and CloudWatch services. You can find the required permissions [here](https://docs.aws.amazon.com/streams/latest/dev/tutorial-stock-data-kplkcl-iam.html). - Access to the Amazon Kinesis target resource - Anypoint Studio or Anypoint Code Builder ## Common Use Cases for the Connector Common use cases for Kinesis Data Streams connector include the following: - Troubleshooting Collect log and event data from sources such as servers, desktops, and mobile devices. Then continuously process the data, generate metrics, power live dashboards, and put the aggregated data into data stores such as Amazon S3. - Real-time analytics Run real-time analytics on high-frequency event data such as sensor data or social media data collected by Amazon Kinesis Data Streams. These analytics enable you to gain insights from your data at a frequency of minutes instead of a frequency of hours or days. - Mobile data capture Have your mobile applications push data to Amazon Kinesis data streams from hundreds of thousands of devices, making the data available to you as soon as it is produced on the mobile devices. For examples, see [Examples](https://docs.mulesoft.com/amazon-kinesis-connector/latest/amazon-kinesis-connector-examples). ## Audience - New users, read: - [Using Anypoint Studio to Configure the Connector](https://docs.mulesoft.com/amazon-kinesis-connector/latest/amazon-kinesis-connector-studio) to create your Mule app - [Examples](https://docs.mulesoft.com/amazon-kinesis-connector/latest/amazon-kinesis-connector-examples), which provide one or more use cases for the connector - Advanced users, read: - [XML and Maven Support](https://docs.mulesoft.com/amazon-kinesis-connector/latest/amazon-kinesis-connector-xml-maven) - [Examples](https://docs.mulesoft.com/amazon-kinesis-connector/latest/amazon-kinesis-connector-examples), which provides one or more use cases for the connector ## Authentication Types Amazon Kinesis Data Streams Connector supports Access Key authentication. With this type of authentication, the connector authenticates to AWS by using an access key, secret key, and optionally, a role. For information about configuring authentication, see [Using Anypoint Studio to Configure Amazon Kinesis Data Streams](https://docs.mulesoft.com/amazon-kinesis-connector/latest/amazon-kinesis-connector-studio). ## Using Exchange Templates and Examples [Anypoint Exchange](https://www.mulesoft.com/exchange/) provides templates that you can use as starting points for your apps and examples that illustrate a complete solution. ## Next Step After you complete the prerequisites and try the templates and examples, you are ready to create your own app and configure the connector using [Anypoint Studio](https://docs.mulesoft.com/amazon-kinesis-connector/latest/amazon-kinesis-connector-studio). ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Use Exchange to Discover Connectors, Templates, and Examples](https://docs.mulesoft.com/connectors/introduction/intro-use-exchange) - [Salesforce Help](https://help.salesforce.com) --- product: Amazon Kinesis Connector version: 2.0 is-latest-version: true --- # Amazon Kinesis Data Streams Connector 2.0 Examples > For the full documentation index, see: https://docs.mulesoft.com/llms.txt The following example shows how to use Anypoint Connector for Amazon Kinesis Data Streams (Amazon Kinesis Data Streams Connector) to put data records and listen from an Amazon Kinesis data stream. > [!NOTE] Amazon Kinesis Data Streams Connector supports > [Enhanced Fan-Out (EFO)](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html) > by default. This means that the connector automatically subscribes to all shards of the configured > stream using EFO. > > The connector’s behavior aligns with AWS Kinesis Client Library (KCL) version 2.0 and later. The > connector uses the same implementation as shown in > [Develop enhanced fan-out consumers using KCL 2.x in Java](https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-kcl-java.html). Before you try the example, access Anypoint Studio (Studio) and verify that the **Mule Palette** view displays entries for Kinesis. If not, follow the instructions in [Add the Connector to Your Project](https://docs.mulesoft.com/amazon-kinesis-connector/latest/amazon-kinesis-connector-studio#add-connector-to-project). ## Flows for This Example The following screenshots show the Anypoint Studio app flows for this example: - This flow uses the **Put Records** operation to put data into the Amazon Kinesis data stream. In this flow, **HTTP > Listener** receives a query parameter named **data** that sets the payload for the **Put Record** operation. It uses a query parameter named **partitionKey** to set the partition key value for the **Put Record** operation. ![Put Record operation flow](/mulesoft/current/en-us/amazon-kinesis-connector/latest/_images/amazon-kinesis-example-put-record.png) - This flow uses the **Listener** source to listen for new data records: ![Listener source flow](/mulesoft/current/en-us/amazon-kinesis-connector/latest/_images/amazon-kinesis-example-listener.png) - This flow uses the **Listener** source to listen for new data records. After receiving the new records, it calls the **Checkpoint** operation to set a manual checkpoint: ![Checkpoint operation flow](/mulesoft/current/en-us/amazon-kinesis-connector/latest/_images/amazon-kinesis-example-checkpoint.png) ## XML Code for This Example ```xml ``` ### See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Amazon Kinesis Connector version: 2.0 is-latest-version: true --- # Amazon Kinesis Data Streams Connector 2.0 Reference > For the full documentation index, see: https://docs.mulesoft.com/llms.txt > [!NOTE] Amazon Kinesis Data Streams Connector doesn’t support Windows because it uses the Kinesis > Producer Library (KPL), which deprecated Windows support starting in version 0.14.0. ## Configurations --- ### Config The configuration for the connector. You can use this configuration for both the producer and consumer. #### Parameters
NameTypeDescriptionDefault ValueRequired

Name

String

The name for this configuration. Connectors reference the configuration with this name.

x

Connection

The connection types to provide to this configuration.

x

Name

String

The identifier of this element used to reference it in other components.

x

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before the runtime considers it eligible for expiration. This does not mean that the platform will expire the instance at the exact moment that it becomes eligible. The runtime will actually purge the instances when it sees it fit.

Aggregation Enabled

Boolean

When set to true, this option packs multiple user data records into a single Amazon Kinesis data record.

true

Response Timeout

Number

The maximum total time that can elapse between the start of an HTTP request and when the responses are received. If this time is exceeded, the request times out.

30

Request Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

A time unit that qualifies the Response Timeout value

SECONDS

Record Max Buffered Time

Number

Maximum amount of time that an Amazon Kinesis data record spends buffering before it is sent. Records can be sent earlier than this value, depending on other buffering limits.

100

Record Max Buffered Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

A time unit that qualifies the Record Max Buffered Time value

MILLISECONDS

Aggregation Max Count

Number

Maximum number of items to pack into an aggregated Amazon Kinesis data record.

4294967295

Aggregation Max Size

Number

Maximum data size to pack into an aggregated Amazon Kinesis data record.

51200

Aggregation Max Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

A data unit that qualifies the Aggregation Max Size value

BYTE

Collection Max Count

Number

Maximum number of items to pack into a batch request.

500

Collection Max Size

Number

Maximum amount of data to send with a Put Record operation.

5242880

Collection Max Size Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

A data unit that qualifies the Collection Max Size value

BYTE

Connection Timeout

Number

The amount of time to wait when initially establishing the TCP connection between the connector and Amazon Kinesis server before throwing an exception if the connection fails

30

Connection Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The time unit for the Connection Timeout value

SECONDS

Max Connections

Number

Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Setting this value too high can impact latency and consume additional resources without increasing throughput.

24

Min Connections

Number

Minimum number of connections to keep open to the backend. In most cases, there is no need to increase value.

1

Rate Limit

Number

Limits the number of data records per second and number of bytes per second sent to a shard. This limit can reduce bandwidth and CPU cycle wastage from sending requests that will fail from throttling. The default value, 150%, enables a single producer instance to completely saturate the allowance for a shard. To reduce throttling errors rather than completely saturate the shard, consider reducing this setting.

150%

Record TTL

Number

Time limit for data records to be put. Records that are not successfully put within this limit fail. When setting this value, take into consideration the fact that data records still incur network latency after they leave the Amazon Kinesis Producer Library (KPL).

30

Record TTL Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The time unit for the Record TTL value.

SECONDS

Thread Pool Size

Number

Maximum number of threads with which to configure the native process thread pool. The default value, 0, enables the KPL process to choose the size of the thread pool.

0

#### Connection Types ##### Kinesis Connection Connects to AWS Kinesis through the use of a user-specified access and secret key and, optionally, a role. These credentials securely sign requests sent to AWS services. ###### Parameters
NameTypeDescriptionDefault ValueRequired

Custom Kinesis Endpoint

String

Specifies a custom Kinesis endpoint, often used for FIPS compliance.

Custom STS endpoint

String

Sets a custom STS endpoint. This is useful when a non-standard service endpoint is required, such as a VPC endpoint or a FIPS-compliant endpoint.

Proxy Configuration

Proxy connection settings for outbound connections. This setting applies to the Listener source and Checkpoint operation. It does not affect the Put Record operation.

TLS Configuration

Protocol to use for communication. When using the HTTPS protocol, the HTTP communication is secured using TLS or SSL. If HTTPS is configured as the protocol, then, at a minimum, you must configure the keystore in the tls:context child element of the listener-config.

This setting applies to the Listener source and Checkpoint operation. It does not affect the Put Record operation.

Access Key

String

Access key provided by Amazon.

x

Secret Key

String

Secret key provided by Amazon.

x

Region Endpoint

String

Topic region endpoint

us-east-1

Role

Role configuration

Custom Dynamo DB Endpoint

String

Sets a Dynamo DB endpoint to track and maintain state information.

Custom Cloud Watch Endpoint

String

Sets a Cloud Watch endpoint for monitoring and observability.

Connection Timeout

Number

The amount of time to wait when initially establishing the TCP connection between the connector and Amazon Kinesis server before throwing an exception if the connection fails

30

Connection Timeout Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

The time unit for the Connection Timeout field.

SECONDS

Response Timeout

Number

The maximum time to wait between the time an HTTP request is sent and when a response is received

30

Response Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Response Timeout value.

SECONDS

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.

## Supported Operations - [Checkpoint](#Checkpoint) - [Put Record](#PutRecord) ## Associated Sources - [Listener](#Listener) ## Operations ## Checkpoint `` The operation checkpoint position of a listener into the Amazon DynamoDB table. Upon failover, the connector starts fetching records that are located after the check pointed position. The **Checkpoint** operation does not modify the current position of the listener. ### Parameters
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Stream Name

String

The stream name.

x

Application Name

String

Name of the DynamoDB table that holds data about the current stream positions.

x

Shard ID

String

The shard identifier.

Sequence Number

String

A sequence number at which to checkpoint in the shard.

Subsequence Number

Number

A subsequence number at which to checkpoint within the shard.

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component

x

Error Mappings

Array of Error Mapping

Set of error mappings.

### For Configurations - [Config](#Config) ### Throws - KINESIS:ILLEGAL\_STATE - KINESIS:INVALID\_ARGUMENT - KINESIS:PROVISIONED\_THROUGHPUT\_EXCEEDED - KINESIS:TIMEOUT - KINESIS:UNAUTHORIZED ## Put Record `` Puts a data record into an Amazon Kinesis data stream. ### Parameters
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Stream Name

String

The stream name

x

Partition Key

String

The partition key. If not provided, the connector generates a UUID for this value.

Explicit Hash Key

String

Overrides the Amazon Kinesis Data Streams explicitHashKey value, which is normally computed with using MD5 function of the data record partition key.

Data

Any

The content to put in the Amazon Kinesis data stream. This content can be up to 1 MB in size.

#[payload]

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component

x

Streaming Strategy

Disables the repeatable stream functionality and uses non-repeatable streams to have lower performance overhead, memory use, and costs

Target Variable

String

Name of the variable in which to store the operation’s output

Target Value

String

Expression that evaluates the operation’s output. The expression outcome is stored in the target variable.

#[payload]

Error Mappings

Array of Error Mapping

Set of error mappings.

Reconnection Strategy

A retry strategy in case of connectivity errors

### Output

Type

Any

### For Configurations - [Config](#Config) ### Throws - KINESIS:CONNECTIVITY - KINESIS:INVALID\_ARGUMENT - KINESIS:PROCESSING - KINESIS:RETRY\_EXHAUSTED - KINESIS:TIMEOUT - KINESIS:UNAUTHORIZED ## Sources ## Listener `` Uses the Amazon Kinesis Client Library (KCL) to listen from a specified point in a stream. > [!NOTE] Due to varying batch sizes, checkpoints can occur in the incorrect sequence, potentially > leading to transaction loss in certain cases. Set `maxConcurrency=1` to ensure transactions are > processed sequentially. ### Parameters
NameTypeDescriptionDefault ValueRequired

Configuration

String

The name of the configuration to use.

x

Output Mime Type

String

The MIME type of the payload that this operation outputs.

Output Encoding

String

The encoding of the payload that this operation outputs.

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component

x

Primary Node Only

Boolean

Whether this listener executes on the primary node only when running in a cluster

On Capacity Overload

Enumeration, one of:

  • WAIT

  • DROP

Strategy that Mule applies when the flow receives more messages than it can handle

WAIT

Primary Node Only

Boolean

Set to true to restrict the execution of this source to the primary node when operating in a cluster environment.

Config Ref

ConfigurationProvider

The name of the configuration to be used to execute this component.

x

Redelivery Policy

Policy for processing the redelivery of a message

Stream Name

String

The stream name.

x

Application Name

String

Name of the DynamoDB table that holds data about current stream positions. If the table doesn’t exist, then the connector creates a new one at the initial position.

x

Shard Configs

Array of Shard Config

The shard configuration. If specified, the listener listens only from selected shards.

Max Batch Size

Number

Maximum number of data records a batch can carry. A batch is the number of records that the listener receives from the stream.

10000

Max Leases For Worker

Number

Number of shards to which the listener can subscribe in parallel.

2137483647

Shard Prioritization

Enumeration, one of:

  • PARENTS_FIRST_SHARD_PRIORITIZATION

  • NO_OP_SHARD_PRIORITIZATION

Logic used to prioritize or filter the shards before their execution.

PARENTS_FIRST_SHARD_PRIORITIZATION Prioritizes parent shards first and limits the number of shards that are available for initialization based on their depth (Max Depth field).

NO_OP_SHARD_PRIORITIZATION Returns the original list of shards without any modifications.

NO_OP_SHARD_PRIORITIZATION

Max Depth

Number

Used by the PARENTS_FIRST_SHARD_PRIORITIZATION shard prioritization option. Any shard that is deeper than this value is excluded from processing.

Read Capacity Units

Number

The maximum number of strongly consistent reads consumed per second before Amazon DynamoDB returns a ThrottlingException.

10

Write Capacity Units

Number

The maximum number of writes consumed per second before Amazon DynamoDB returns a ThrottlingException.

10

Failover Time

Number

Duration of a lease for a worker. This parameter also determines the frequency with which a worker looks for new leases to work on. The frequency is roughly twice the specified value.

10

Failover Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Failover Time field.

SECONDS

Checkpoint On Complete

Boolean

If true, the listener checkpoints the current reading position to Amazon DynamoDB after data records are processed by the owning flow, without taking into account whether the processing finished successfully.

true

Absolute Position

Enumeration, one of:

  • LATEST

  • FIRST

Whether the listener listens from the latest data record or the oldest data record

Timestamp

DateTime

Date and time from which the listener listens

Reconnection Strategy

A retry strategy in case of connectivity errors

### Output

Type

Array of binary messages with [Record Attributes]

### For Configurations - [Config](#Config) ## Types ### Proxy Configuration
FieldTypeDescriptionDefault ValueRequired

Host

String

Host where the proxy requests are sent.

x

Port

Number

Port where the proxy requests are sent.

x

Username

String

The username to authenticate against the proxy.

Password

String

The password to authenticate against the proxy.

Non Proxy Hosts

Array of String

A list of hosts against which the proxy should not be used.

Ntlm Domain

String

The domain to authenticate against the proxy.

### TLS
FieldTypeDescriptionDefault ValueRequired

Enabled Protocols

String

A comma-separated list of protocols enabled for this context.

Enabled Cipher Suites

String

A comma-separated list of cipher suites enabled for this context.

Trust Store

Key Store

Revocation Check

### Trust Store
FieldTypeDescriptionDefault ValueRequired

Path

String

The location (which is resolved relative to the current classpath and file system, if possible) of the truststore.

Password

String

The password used to protect the truststore.

Type

String

The type of store used.

Algorithm

String

The algorithm used by the truststore.

Insecure

Boolean

If set to true, no certificate validations are performed, which makes connections vulnerable to attacks. Use at your own risk.

### Key Store
FieldTypeDescriptionDefault ValueRequired

Path

String

The location (which is resolved relative to the current classpath and file system, if possible) of the keystore.

Type

String

The type of store used.

Alias

String

When the keystore contains many private keys, this attribute indicates the alias of the key that should be used. If not defined, the first key in the file is used by default.

Key Password

String

The password used to protect the private key.

Password

String

The password used to protect the keystore.

Algorithm

String

The algorithm used by the keystore.

### Standard Revocation Check
FieldTypeDescriptionDefault ValueRequired

Only End Entities

Boolean

Verifies the last element of the certificate chain only.

Prefer Crls

Boolean

Tries CRL instead of OCSP first.

No Fallback

Boolean

Does not use the secondary checking method.

Soft Fail

Boolean

Avoids a certification failure when the revocation server cannot be reached or is busy.

### Custom Ocsp Responder
FieldTypeDescriptionDefault ValueRequired

Url

String

The URL of the OCSP responder.

Cert Alias

String

Alias of the signing certificate for the OCSP response (must be in the truststore), if present.

### Crl File
FieldTypeDescriptionDefault ValueRequired

Path

String

The path to the CRL file.

### Role
FieldTypeDescriptionDefault ValueRequired

Arn

String

The Amazon Resource Name (ARN) of the role to assume.

x

External Id

String

A unique identifier that might be required when you assume a role in another account. If the administrator of the account to which the role belongs provides an external ID, then provide that value in this field.

Duration

Number

The duration of the role session.

3600

Duration Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Duration field

SECONDS

Referred Policy Arns

Array of String

The Amazon Resource Names (ARNs) of the IAM-managed policies to use as managed session policies. The policies must exist in the same account as the role.

Tags

Object

A list of session tags that you want to pass. Each session tag consists of a key name and an associated value.

### Reconnection
FieldTypeDescriptionDefault ValueRequired

Fails Deployment

Boolean

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.

Reconnection Strategy

The reconnection strategy to use.

### Reconnect
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often in milliseconds to reconnect

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

Count

Number

How many reconnection attempts to make.

### Reconnect Forever
FieldTypeDescriptionDefault ValueRequired

Frequency

Number

How often in milliseconds to reconnect

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

### Expiration Policy
FieldTypeDescriptionDefault ValueRequired

Max Idle Time

Number

A scalar time value for the maximum amount of time a dynamic configuration instance should be allowed to be idle before it’s considered eligible for expiration.

Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

A time unit that qualifies the Max Idle Time attribute.

### Record Attributes
FieldTypeDescriptionDefault ValueRequired

Aggregated

Boolean

Approximate Arrival Timestamp

String

Encryption Type

String

Partition Key

String

Sequence Number

String

Shard Id

String

Sub Sequence Number

Number

### Redelivery Policy
FieldTypeDescriptionDefault ValueRequired

Max Redelivery Count

Number

The maximum number of times a message can be redelivered and processed unsuccessfully before triggering a process-failed message.

Message Digest Algorithm

String

The secure hashing algorithm to use.

SHA-256

Message Identifier

The strategy used to identify the messages.

Object Store

ObjectStore

The object store that stores the redelivery counter for each message.

### Redelivery Policy Message Identifier
FieldTypeDescriptionDefault ValueRequired

Use Secure Hash

Boolean

Whether to use a secure hash algorithm to identify a redelivered message.

Id Expression

String

One or more expressions to use to determine when a message was redelivered. This property can be set only if the value of the Use Secure Hash field is false.

### Shard Config
FieldTypeDescriptionDefault ValueRequired

Shard Id

String

The shard identifier.

x

Shard Initial Position

The shard initial position.

x

### Shard Initial Position
FieldTypeDescriptionDefault ValueRequired

At Sequence Number

String

The listener starts listening at this position in the data stream.

After Sequence Number

String

The listener starts listening after this position in the data stream.

Absolute Position

Enumeration, one of:

  • LATEST

  • FIRST

The listener starts listening from the latest or oldest record, depending on the value.

Timestamp

DateTime

### Error Mapping
FieldTypeDescriptionDefault ValueRequired

Source

Enumeration, one of:

  • ANY

  • REDELIVERY_EXHAUSTED

  • TRANSFORMATION

  • EXPRESSION

  • SECURITY

  • CLIENT_SECURITY

  • SERVER_SECURITY

  • ROUTING

  • CONNECTIVITY

  • RETRY_EXHAUSTED

  • TIMEOUT

Target

String

x

### Repeatable In Memory Stream
FieldTypeDescriptionDefault ValueRequired

Initial Buffer Size

Number

The amount of memory to allocate to consume the stream and provide random access to it. If the stream contains more data than can fit into this buffer, then the buffer expands according to the Buffer Size Increment value, with an upper limit of the Max In Memory Size value.

Buffer Size Increment

Number

This is by how much the buffer size expands if it exceeds its initial size. Setting a value of zero or lower means that the buffer should not expand, meaning that a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised when the buffer is full.

Max Buffer Size

Number

The maximum amount of memory to use. If more memory is used, then a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value less than or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit in which the fields in this table are expressed

### Repeatable File Store Stream
FieldTypeDescriptionDefault ValueRequired

In Memory Size

Number

Maximum memory that the stream should use to keep data in memory. If more than that is consumed, content on the disk is buffered.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

The unit in which the In Memory Size value is expressed

## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Reference](#) - [Salesforce Help](https://help.salesforce.com) --- product: Amazon Kinesis Connector version: 2.0 is-latest-version: true --- # Amazon Kinesis Data Streams Connector 2.0 XML and Maven Support > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Although you can manually code a Mule app in XML, it is more efficient to use Anypoint Studio. If you manually code a Mule runtime engine (Mule) app in XML either from the Anypoint Studio XML editor or from a text editor, you can access the connector from your app by adding reference statements to both your XML Mule flow and the Apache Maven `pom.xml` file. When you add the connector in Studio, Studio automatically populates the XML code with the connector’s namespace and schema location and it also adds a dependency to the `pom.xml` file. ## Add a Namespace for the Connector Paste the following code inside the `mule` tag of the header of your configuration XML: ```xml http://www.mulesoft.org/schema/mule/amazon-kinesis http://www.mulesoft.org/schema/mule/amazon-kinesis/current/mule-amazon-kinesis.xsd ``` This example shows how the namespace and schema statements are placed in the XML: ```xml com.mulesoft.connectors amazon-kinesis-data-streams-connector x.x.x mule-plugin ``` Replace `x.x.x` with the version that corresponds to the connector you are using. To obtain the most up-to-date `pom.xml` file information: 1. Go to [Anypoint Exchange](https://www.mulesoft.com/exchange/). 2. In Exchange, click **Login** and supply your Anypoint Platform username and password. 3. In Exchange, search for `amazon kinesis`. 4. Select the connector. 5. Click **Dependency Snippets** near the upper right of the screen. ## Next Step After completing your namespace and `pom.xml` file, you can try the [Examples](https://docs.mulesoft.com/amazon-kinesis-connector/latest/amazon-kinesis-connector-examples), which provides one or more use cases for the connector. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Amazon Kinesis Connector version: 2.0 is-latest-version: true --- # Upgrading and Migrating Amazon Kinesis Data Streams Connector to Version 2.x > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Upgrade Anypoint Connector for Amazon Kinesis Data Streams (Amazon Kinesis Data Streams Connector) to version 2.x. ## Supported Upgrade Paths
From VersionTo Version

1.x

2.x

## Changes in This Release - The `kinesis-client` library is upgraded to version 3.2.0. - The `kinesis-producer` library is upgraded to version 1.0.6. - The `aws-sts` library is upgraded to version 2.38.3. - The `aws-sdk` library is upgraded to version 2.38.3. - The `netty-codec-http` library is upgraded to version 4.2.7.Final. - The `netty-codec-http2` library is upgraded to version 4.2.7.Final. - The `netty-codec` library is upgraded to version 4.2.7.Final. - The `kafka-clients` library is upgraded to version 4.1.0. - The `protobuf-java` library is upgraded to version 4.33.0. - The `commons-lang3` library is upgraded to version 3.19.0. ## Requirements Ensure your system meets the following compatibility requirements before you perform the upgrade:
Application/ServiceVersion

Mule

4.3.0 and later

OpenJDK

8, 11, and 17

Amazon-Kinesis-Client

3.2.0

Amazon-Kinesis-Producer

1.0.6

## Upgrade Prerequisites Before you perform the upgrade, create a backup of your files, data, and configuration in case you need to restore to the previous version. ## Upgrade Steps Follow these steps to perform the upgrade to version 2.x: 1. In Anypoint Studio, create a Mule project. 2. In the **Mule Palette** view, click **Search in Exchange**. 3. In **Add Dependencies to Project**, enter `amazon kinesis data streams` in the search field. 4. In **Available modules**, select **Amazon Kinesis Data Streams** and click **Add**. 5. Click **Finish**. 6. Verify that the Amazon Kinesis Data Streams dependency version is 2.x in the `pom.xml` file in the Mule project. Studio upgrades the connector automatically. ## Post-Upgrade Steps After you install the latest version of the connector, verify that your error handling is updated to work with the minor changes in error codes in this version of the connector. ## Verify the Upgrade After you install the latest version of the connector, follow these steps to verify the upgrade: 1. In Studio, verify that there are no errors in the **Problems** or **Console** views. 2. Verify that there are no problems in the project `pom.xml` file. 3. Test the connection to verify that the operations work. ## Revert the Upgrade If it is necessary to revert to the previous version of Amazon Kinesis Data Streams, change the `mule-amazon-kinesis` dependency version in the project’s `pom.xml` file to the previous version. You must update the project’s `pom.xml` file in Anypoint Studio. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Salesforce Help](https://help.salesforce.com) --- product: Amazon Kinesis Connector version: 1.0 --- # Using Anypoint Studio to Configure Amazon Kinesis Data Streams Connector 1.0 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Studio (Studio) editors help you design and update your Mule applications, properties, and configuration files. To add and configure a connector in Studio: 1. [Create a mule project](#create-mule-project). 2. [Add the connector to your Mule project](#add-connector-to-project). 3. [Add a connector operation to the flow](#add-connector-operation). 4. [Configure a global element for the connector](#configure-global-element) When you run the connector, you can view the app log to check for problems, as described in [View the App Log](#view-app-log). If you are new to configuring connectors in Studio, see [Using Anypoint Studio to Configure a Connector](https://docs.mulesoft.com/connectors/introduction/intro-config-use-studio). ## Create a Mule Project In Studio, create a new Mule project in which to add and configure the connector: 1. In Studio, select **File > New > Mule Project**. 2. Enter a name for your Mule project and click **Finish**. ## Add the Connector to Your Mule Project Add Anypoint Connector for Amazon Kinesis Data Streams Connector (Amazon Kinesis Data Streams Connector) to your Mule project to automatically populate the XML code with the connector’s namespace and schema location and add the required dependencies to the project’s `pom.xml` file: 1. In the **Mule Palette** view, click **(X) Search in Exchange**. 2. In **Add Modules to Project**, type `kinesis` in the search field. 3. Click **Amazon Kinesis Data Streams Connector** in **Available modules**. 4. Click **Add**. 5. Click **Finish**. Adding a connector to a Mule project in Studio does not make that connector available to other projects in your Studio workspace. ### Configure a Source A source initiates a flow when a specified condition is met. You can configure one of these sources to use with Amazon Kinesis Data Streams Connector: - **Listener** Enables you to subscribe to a particular data stream. This source uses the Kinesis Client Library (KCL) to listen from a specified initial point in the data stream. It returns dynamically sized batches of records, however, the max size can also be configured. - **HTTP > Listener** Initiates a flow each time it receives a request on the configured host and port - **Scheduler** Initiates a flow when a time-based condition is met For example, to configure the Amazon Kinesis **Listener** source, follow these steps: 1. In the **Mule Palette** view, select **Kinesis**. 2. Drag **Listener** to the Studio canvas. 3. On the **Listener** properties window, optionally change the value of the **Display Name** field. 4. Click the plus sign (**+**) next to the **Connector configuration** field to configure a global element that can be used by all instances of the Amazon Kinesis Connector in the app. 5. On the **General** tab, complete the following fields:
FieldDescription

Name

Name used to reference this connector instance

Access Key

Access key provided by Amazon

Secret Key

Secret key provided by Amazon

Region Endpoint

URL entry point for a web service

Role

IAM role, if any

6. On the **Security** tab, optionally specify the TLS information for the connector. 7. On the **Advanced** tab, optionally specify reconnection information, including a reconnection strategy. 8. Click **Test Connection** to confirm that Mule can connect with the specified server. 9. Click **OK**. 10. In the **Listener Params** section of the **Listener** properties window, optionally enter values for the fields. ## Add a Connector Operation to the Flow When you add a connector operation to your flow, you immediately define a specific operation for that connector to perform. You can use the following operations: - **Checkpoint**, which keeps track of the current position in the Kinesis data stream - **Seek**, which moves the reading position of the sources (listeners) While moving forward in the stream is fluent, moving backward causes an internal restart of the Kinesis Client Library (KCL) instance. Therefore, use this operation sparingly to move backward in a stream. - **Put Record**, which puts a record into the Kinesis data stream. If the **Aggregation Enabled** field in the global configuration **Advanced** tab is `true` (the default), then multiple records are combined and stored in one Kinesis Data Streams record. Records include binary data, a partition key, and an explicit hash key. The explicit hash key value is normally computed with the MD5 function of the record partition key, but you can override the value by using the **Explicit Hash Key** field. The operation uses the Kinesis Producer Library (KPL), which increases throughput. To add an operation for Amazon Kinesis Data Streams Connector, follow these steps: 1. In the **Mule Palette** view, select **Kinesis** and then select the desired operation. 2. Drag the operation onto the Studio canvas to the right of the source. ## Configure a Global Element for the Connector When you configure a connector, it’s best to configure a global element that all instances of that connector in the app can use. Configuring a global element requires you to provide the authentication credentials that the connector requires to access the target Amazon Kinesis Data Streams system. Amazon Kinesis Data Streams Connector supports Access Key authentication, which uses **Access Key** and **Secret Key** field values for AWS Identity and Access Management (IAM). When you configure a global element, you can reference a configuration file that contains ANT-style property placeholders (recommended), or you can enter your authorization credentials in the global configuration properties. For information about the benefits of using property placeholders and how to configure them, see [Anypoint Connector Configuration](https://docs.mulesoft.com/connectors/introduction/intro-connector-configuration-overview). To configure a global element, follow these steps: 1. Select the name of the connector in the Studio canvas. 2. In the configuration screen for the operation, click the plus sign (**+**) next to the **Connector configuration** field to access the global element configuration fields. 3. On the **General** tab, configure the following fields.
FieldDescription

Name

Name used to reference the connector instance

Access Key

Access key provided by Amazon

Secret Key

Secret key provided by Amazon

Region Endpoint

URL entry point for a web service

Role

IAM role, if any

4. On the **Advanced** tab, optionally specify proxy and reconnection information, including a reconnection strategy. 5. Click **Test Connection** to confirm that Mule can connect with the specified server. 6. Click **OK**. ## View the App Log To check for problems, you can view the app log as follows: - If you’re running the app from Anypoint Platform, the app log output is visible in the Anypoint Studio console window. - If you’re running the app using Mule from the command line, the app log output is visible in your OS console. Unless the log file path is customized in the app’s log file (`log4j2.xml`), you can also view the app log in the default location `MULE_HOME/logs/.log`. You can configure the location of the log path in the app log file `log4j2.xml`. ## Next Step After you configure a global element and connection information, configure the other fields for the connector. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Reference](https://docs.mulesoft.com/amazon-kinesis-connector/1.0/amazon-kinesis-connector-reference) - [Salesforce Help](https://help.salesforce.com) --- product: Amazon Kinesis Connector version: 2.0 is-latest-version: true --- # Using Anypoint Studio to Configure Amazon Kinesis Data Streams Connector 2.0 > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Studio (Studio) editors help you design and update your Mule applications, properties, and configuration files. To add and configure a connector in Studio: 1. [Create a mule project](#create-mule-project). 2. [Add the connector to your Mule project](#add-connector-to-project). 3. [Add a connector operation to the flow](#add-connector-operation). 4. [Configure a global element for the connector](#configure-global-element) When you run the connector, you can view the app log to check for problems, as described in [View the App Log](#view-app-log). If you are new to configuring connectors in Studio, see [Using Anypoint Studio to Configure a Connector](https://docs.mulesoft.com/connectors/introduction/intro-config-use-studio). ## Create a Mule Project In Studio, create a new Mule project in which to add and configure the connector: 1. In Studio, select **File > New > Mule Project**. 2. Enter a name for your Mule project and click **Finish**. ## Add the Connector to Your Mule Project Add Anypoint Connector for Amazon Kinesis Data Streams Connector (Amazon Kinesis Data Streams Connector) to your Mule project to automatically populate the XML code with the connector’s namespace and schema location and add the required dependencies to the project’s `pom.xml` file: 1. In the **Mule Palette** view, click **(X) Search in Exchange**. 2. In **Add Modules to Project**, type `kinesis` in the search field. 3. Click **Amazon Kinesis Data Streams Connector** in **Available modules**. 4. Click **Add**. 5. Click **Finish**. Adding a connector to a Mule project in Studio does not make that connector available to other projects in your Studio workspace. ### Configure a Source A source initiates a flow when a specified condition is met. You can configure one of these sources to use with Amazon Kinesis Data Streams Connector: - **Listener** Enables you to subscribe to a particular data stream. This source uses the Kinesis Client Library (KCL) to listen from a specified initial point in the data stream. It returns dynamically sized batches of records, however, the max size can also be configured. - **HTTP > Listener** Initiates a flow each time it receives a request on the configured host and port - **Scheduler** Initiates a flow when a time-based condition is met For example, to configure the Amazon Kinesis **Listener** source, follow these steps: 1. In the **Mule Palette** view, select **Kinesis**. 2. Drag **Listener** to the Studio canvas. 3. On the **Listener** properties window, optionally change the value of the **Display Name** field. 4. Click the plus sign (**+**) next to the **Connector configuration** field to configure a global element that can be used by all instances of the Amazon Kinesis Connector in the app. 5. On the **General** tab, complete the following fields:
FieldDescription

Name

Name used to reference this connector instance

Access Key

Access key provided by Amazon

Secret Key

Secret key provided by Amazon

Region Endpoint

URL entry point for a web service

Role

IAM role, if any

6. On the **Security** tab, optionally specify the TLS information for the connector. 7. On the **Advanced** tab, optionally specify reconnection information, including a reconnection strategy. 8. Click **Test Connection** to confirm that Mule can connect with the specified server. 9. Click **OK**. 10. In the **Listener Params** section of the **Listener** properties window, optionally enter values for the fields. ## Add a Connector Operation to the Flow When you add a connector operation to your flow, you immediately define a specific operation for that connector to perform. You can use the following operations: - **Checkpoint**, which keeps track of the current position in the Kinesis data stream - **Seek**, which moves the reading position of the sources (listeners) While moving forward in the stream is fluent, moving backward causes an internal restart of the Kinesis Client Library (KCL) instance. Therefore, use this operation sparingly to move backward in a stream. - **Put Record**, which puts a record into the Kinesis data stream. If the **Aggregation Enabled** field in the global configuration **Advanced** tab is `true` (the default), then multiple records are combined and stored in one Kinesis Data Streams record. Records include binary data, a partition key, and an explicit hash key. The explicit hash key value is normally computed with the MD5 function of the record partition key, but you can override the value by using the **Explicit Hash Key** field. The operation uses the Kinesis Producer Library (KPL), which increases throughput. To add an operation for Amazon Kinesis Data Streams Connector, follow these steps: 1. In the **Mule Palette** view, select **Kinesis** and then select the desired operation. 2. Drag the operation onto the Studio canvas to the right of the source. ## Configure a Global Element for the Connector When you configure a connector, it’s best to configure a global element that all instances of that connector in the app can use. Configuring a global element requires you to provide the authentication credentials that the connector requires to access the target Amazon Kinesis Data Streams system. Amazon Kinesis Data Streams Connector supports Access Key authentication, which uses **Access Key** and **Secret Key** field values for AWS Identity and Access Management (IAM). When you configure a global element, you can reference a configuration file that contains ANT-style property placeholders (recommended), or you can enter your authorization credentials in the global configuration properties. For information about the benefits of using property placeholders and how to configure them, see [Anypoint Connector Configuration](https://docs.mulesoft.com/connectors/introduction/intro-connector-configuration-overview). To configure a global element, follow these steps: 1. Select the name of the connector in the Studio canvas. 2. In the configuration screen for the operation, click the plus sign (**+**) next to the **Connector configuration** field to access the global element configuration fields. 3. On the **General** tab, configure the following fields.
FieldDescription

Name

Name used to reference the connector instance

Access Key

Access key provided by Amazon

Secret Key

Secret key provided by Amazon

Region Endpoint

URL entry point for a web service

Role

IAM role, if any

4. On the **Advanced** tab, optionally specify proxy and reconnection information, including a reconnection strategy. 5. Click **Test Connection** to confirm that Mule can connect with the specified server. 6. Click **OK**. ## View the App Log To check for problems, you can view the app log as follows: - If you’re running the app from Anypoint Platform, the app log output is visible in the Anypoint Studio console window. - If you’re running the app using Mule from the command line, the app log output is visible in your OS console. Unless the log file path is customized in the app’s log file (`log4j2.xml`), you can also view the app log in the default location `MULE_HOME/logs/.log`. You can configure the location of the log path in the app log file `log4j2.xml`. ## Next Step After you configure a global element and connection information, configure the other fields for the connector. ## See Also - [Introduction to Anypoint Connectors](https://docs.mulesoft.com/connectors/introduction/introduction-to-anypoint-connectors) - [Reference](https://docs.mulesoft.com/amazon-kinesis-connector/latest/amazon-kinesis-connector-reference) - [Salesforce Help](https://help.salesforce.com)