---
schema_version: "1"
id: 300ce195070db632b5897abd10d7687f449c0ee8b6497ff8acf5a6910526baa3
title: OpenTelemetry Support in Mule Runtime
collection: mulesoft
version: current
locale: en-us
portal_url: https://docs.salesforce.com/mulesoft/current/en-us/mule-runtime/4.12/otel-support.md
canonical_url: https://docs.mulesoft.com/mule-runtime/4.12/otel-support
source_site: docs.mulesoft.com
content_type: documentation
content_hash: ceda7a8f30a74042b1d6f2f01d637dbaccf06885aed5987fa457fc6f49ae17c1
description: MuleSoft Documentation Site
release: "4.12"
latest: false
---
---
product: Mule Runtime
version: 4.12
---

# OpenTelemetry Support in Mule Runtime

> For the full documentation index, see: https://docs.mulesoft.com/llms.txt

OpenTelemetry is an observability standard consisting of specifications, APIs, and SDKs. It helps
you instrument, generate, collect, and export telemetry data, such as metrics, logs, and traces, to
analyze software behavior.

OpenTelemetry enables Mule runtime engine to provide observability into the behavior of Mule
applications. With tracing context propagation, Mule also brings observability to interactions
between Mule apps and non-Mule components that leverage this standard.

Mule supports generating and exporting OpenTelemetry traces, logs, and metrics.

## Before You Begin

To use the OpenTelemetry Mule features, you must be familiar with:

- Distributed tracing concepts, including traces and spans.

  If you’re new to distributed tracing and OpenTelemetry, we recommend starting with
  [What is OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) and
  [Understanding distributed tracing](https://opentelemetry.io/docs/concepts/observability-primer/#understanding-distributed-tracing).

- [How to export telemetry data to third-party monitoring systems](https://docs.mulesoft.com/monitoring/telemetry-exporter).

Before starting with the observability standard, you must have:

- For traces and logs, Mule runtime 4.11.0 or later.

- For metrics, Mule runtime 4.12.0 or later.

- For OpenTelemetry tracing context propagation, Anypoint Connector for HTTP (HTTP Connector) 1.8 or
  later.

- Mule runtime Direct Telemetry Stream.

> [!NOTE] Direct Telemetry Stream is available only for Advanced and Titanium tiers of MuleSoft
> Anypoint Platform.

## Tracing a Mule Application

Mule leverages OpenTelemetry to provide:

- Application execution observability

  Tracing a Mule runtime application allows you to observe its execution while maintaining the
  mental model of a Mule engineer. Its main objective is to clearly show how the Mule application
  processes messages, helping you to develop, monitor, and troubleshoot more effectively.

- Distributed tracing

  Tracing also enables you to track Mule application interactions. When multiple systems and
  services are involved, distributed tracing tracks requests as they flow through distributed
  environments, providing a comprehensive view of the overall execution.

Distributed tracing is achieved via the tracing context propagation. For example, if an HTTP
Listener receives [W3C Trace Context](https://www.w3.org/TR/trace-context/) headers, the flow span
in Mule acquires the client’s remote parent span, the span where the HTTP call originated. Mule
automatically propagates the current span to the server endpoints when using the HTTP Requester
component.

The span context format supported for receiving and propagating the context is the
[W3C Trace Context](https://www.w3.org/TR/trace-context/) format.

### Mule Span Data

A trace is a sequence of spans representing a complete operation in a distributed system. Spans can
be nested and represent a unit of work or operation. Each span includes:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Categories</th><th>Description</th><th>Example Values</th></tr></thead><tbody><tr><td><div><div><p>Name</p></div></div></td><td><div><div><p>Name of the span</p></div></div></td><td><div><div><p><code>mule:set-variable</code></p></div></div></td></tr><tr><td><div><div><p>Parent Span ID</p></div></div></td><td><div><div><p>Span ID of current span’s parent</p></div></div></td><td><div><div><p><code>86838830d494d679</code></p></div></div></td></tr><tr><td><div><div><p>Start and End Timestamps</p></div></div></td><td><div><div><p>Earliest and latest timestamp for a span</p></div></div></td><td><div><div><p><code>2021-10-22 16:04:01.209458162 +0000 UTC</code></p></div></div></td></tr><tr><td><div><div><p>Span Context</p></div></div></td><td><div><div><p>Data that uniquely identifies the span: trace ID, span ID, trace flags, and trace state</p></div></div></td><td><div></div></td></tr><tr><td><div><div><p>Attributes</p></div></div></td><td><div><div><p>Key-value pairs containing metadata used to annotate a span to carry information about the operation it’s tracking.</p></div></div></td><td><div><div><p>{ “location”: “flow/processors/2”, “correlation.id”: “1234abcd” }</p></div></div></td></tr><tr><td><div><div><p>Span Events</p></div></div></td><td><div><div><p>A structured log message or annotation on a span, used to denote a meaningful, unique point in time during the span’s duration.</p></div></div></td><td><div><div><p>{ "name": "", "message": "OK", "timestamp": "2021-10-22 16:04:01.209512872 +0000 UTC" }</p></div></div></td></tr><tr><td><div><div><p>Span Links</p></div></div></td><td><div><div><p>Links that associate one span with one or more spans, indicating a causal relationship.</p></div></div></td><td><div></div></td></tr><tr><td><div><div><p>Span Status</p></div></div></td><td><div><div><p>Status attached to a span. You can set a span status when a known error, such as an exception, occurs in the app code.</p></div></div></td><td><div><div><p><code>ERROR</code>, <code>OK</code>, <code>UNSET</code></p></div></div></td></tr><tr><td><div><div><p>Span Kind</p></div></div></td><td><div><div><p>Information about how the trace is assembled.</p></div></div></td><td><div><div><p><code>CLIENT</code>, <code>INTERNAL</code>, <code>PRODUCER</code></p></div></div></td></tr></tbody></table>

### Example of Flow Tracing Feature

When tracing a flow execution, each Mule component execution involved in processing the message is
represented as a span. The component spans describe the relevant aspects of the execution. They can
range from a single span for a simple component, such as the Set Payload, to more complex
structures, such as a Batch Job.

Batch Instance

The Mule runtime traces complex components, such as Batch Jobs.

![A batch flow including Listener and Set Payload components, and a more complex structure](/mulesoft/current/en-us/mule-runtime/4.12/_images/otel-batch-instance.png)

Spans Generated

The trace shows how the first step of the Batch Job processes each batch record, with each record
span containing a child span that represents the unit of work the Logger component performs. Next,
it displays the Batch Aggregator span, along with the For Each component and the processors within
it, such as Transform Message and Logger. Then, the trace displays the other two batch steps and
their respective components. Finally, it shows the Batch On Complete span, along with the Logger
component that appears after the Batch Job in the flow.

### Example of Tracing Context Propagation Feature

In this example, one Mule app sends a request to another Mule app:

![A flow including a Listener, Logger, and Request components of an HTTP app](/mulesoft/current/en-us/mule-runtime/4.12/_images/otel-mule-app1.png)

![A flow showing the process flow for an asynchronous application](/mulesoft/current/en-us/mule-runtime/4.12/_images/otel-mule-app2.png)

Distributed tracing captures both flows as part of the same trace because OpenTelemetry is
instrumented in Mule and HTTP Connector.

### Mule Errors in Tracing

If the execution of the unit of work represented by a span produces an error, the span’s status is
set to `Error`. MuleSoft follows the semantic convention defined by OpenTelemetry that adds one or
more `exception` events to the span:

- `exception.type`: Mule error type

- `exception.message`: Mule error message

- `exception.stacktrace`: Mule flow stack string representation

In this example, when an error is produced, the span corresponding to the Raise Error component has
an `Error` status:

![A flow of a telemetry flow with error handling](/mulesoft/current/en-us/mule-runtime/4.12/_images/otel-errors-in-tracing.png)

Because this flow lacks error handling, the error propagates and fails the flow execution, causing
the `Error` status to propagate to the flow span. A detailed view of the span also shows the
associated exception events.

### HTTP Conventions

The HTTP Listener and HTTP Request operations contribute additional attributes to the spans,
providing more context for tracing the HTTP interactions.

For flows with an HTTP Listener:

- Span name: Combination of HTTP Method and path. For example, `POST /api`.

- Span kind: `SERVER`.

Additional attributes for an HTTP Listener associated flow span:

<table><colgroup><col> <col></colgroup><thead><tr><th>Attribute</th><th>Description</th></tr></thead><tbody><tr><td><p><code>net.host.name</code></p></td><td><p>Host address of the HTTP Listener</p></td></tr><tr><td><p><code>net.host.port</code></p></td><td><p>Port of the HTTP Listener</p></td></tr><tr><td><p><code>http.user_agent</code></p></td><td><p>User agent of the received request</p></td></tr><tr><td><p><code>http.route</code></p></td><td><p>Path of the HTTP Listener</p></td></tr><tr><td><p><code>http.method</code></p></td><td><p>HTTP method</p></td></tr><tr><td><p><code>http.flavor</code></p></td><td><p>HTTP version</p></td></tr></tbody></table>

For flows with an HTTP Request:

- Span name: HTTP Method. For example, `GET`.

- Span kind: `CLIENT`.

Additional attributes for an HTTP Request associated flow span:

<table><colgroup><col> <col></colgroup><thead><tr><th>Attribute</th><th>Description</th></tr></thead><tbody><tr><td><p><code>http.url</code></p></td><td><p>Target URL</p></td></tr><tr><td><p><code>net.host.port</code></p></td><td><p>Port of the HTTP Request</p></td></tr><tr><td><p><code>net.peer.name</code></p></td><td><p>Target IP address</p></td></tr><tr><td><p><code>http.route</code></p></td><td><p>Path of the HTTP Request</p></td></tr><tr><td><p><code>http.flavor</code></p></td><td><p>HTTP version</p></td></tr></tbody></table>

## OpenTelemetry Logs

Mule runtime implements its [logging](https://docs.mulesoft.com/mule-runtime/4.12/logging-in-mule) by leveraging Log4j, where you can configure
appenders and logging levels to send logs to different destinations.

The OpenTelemetry logging export is implemented along with Log4j logging. This means you can
configure OpenTelemetry logging export on top of your existing logging setup. This flexible
architecture doesn’t force a full transition to OpenTelemetry.

Exported logs include application, domain, and Mule runtime server logs.

### Mule Log Record Data

OpenTelemetry logging is a sequence of log records with a unified data model, where each record
corresponds to a Java log call. Each log record contains:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Categories</th><th>Description</th><th>Example Values</th></tr></thead><tbody><tr><td><div><div><p>Timestamp</p></div></div></td><td><div><div><p>Time of the Java log call</p></div></div></td><td><div><div><p><code>2021-10-22 16:04:01.209458162 +0000 UTC</code></p></div></div></td></tr><tr><td><div><div><p>Observed Timestamp</p></div></div></td><td><div><div><p>Time of the OpenTelemetry log event registration</p></div></div></td><td><div><div><p><code>2021-10-22 16:04:01.209458162 +0000 UTC</code></p></div></div></td></tr><tr><td><div><div><p>Trace ID</p></div></div></td><td><div><div><p>Trace where the log event originated. Not present if the log occurred outside an application trace.</p></div></div></td><td><div><div><p><code>4a7b9e1d5c3f8a0e2b6d1f7c8a9e0d2c</code></p></div></div></td></tr><tr><td><div><div><p>Span ID</p></div></div></td><td><div><div><p>Span where the log event originated. Not present if the log occurred outside an application trace.</p></div></div></td><td><div><div><p><code>4b7f3c798460</code></p></div></div></td></tr><tr><td><div><div><p>Trace Flags</p></div></div></td><td><div><div><p>W3C trace flags corresponding to the trace where the log event originated. Not present if the log occurred outside an application trace.</p></div></div></td><td><div><div><p><code>01x</code></p></div></div></td></tr><tr><td><div><div><p>Severity Text</p></div></div></td><td><div><div><p>Severity of the log event. Mule maps this value to the equivalent Log4j level during export.</p></div></div></td><td><div><div><p><code>INFO</code></p></div></div></td></tr><tr><td><div><div><p>Body</p></div></div></td><td><div><div><p>Body of the log record. Also known as log message.</p></div></div></td><td><div><div><p><code>Stopping flow: test-flow</code></p></div></div></td></tr><tr><td><div><div><p>Instrumentation Scope</p></div></div></td><td><div><div><p>Fully qualified name of the Java class where the Java log call originated</p></div></div></td><td><div><div><p><code>org.mule.runtime.core.internal.lifecycle.AbstractLifecycleManager</code></p></div></div></td></tr><tr><td><div><div><p>Attributes</p></div></div></td><td><div><div><p>Key-value pairs that annotate a log record with context metadata. Includes Log4j MDC entries.</p></div></div></td><td><div><div><p><code>{ “location”: “flow/processors/2”, “correlation.id”: “1234abcd” }</code></p></div></div></td></tr></tbody></table>

### MDC Propagation

Any entry added to the Log4j MDC is added to the OpenTelemetry logs as Log Record attributes, so all
Log4j MDC data is exported as part of OpenTelemetry logs.

For adding custom attributes on both traces and logs, see [Custom Attributes](#custom-attributes).

### Correlation with Traces

- When using OpenTelemetry logs:

  OpenTelemetry Log Records include fields for the trace ID and span ID. When tracing is enabled,
  Mule runtime exports OpenTelemetry logs that include these values, enabling correlation between
  application traces and logs.

- When using Log4j logs:

  If you export traces only, Mule runtime still adds the trace ID and span ID to the
  [Log4j MDC](https://docs.mulesoft.com/mule-runtime/4.12/logging-mdc) under the keys `trace_id` and `span_id`.

## OpenTelemetry Metrics

Metrics provide visibility into the health and operational performance of the Mule runtime, Mule
applications and the Direct Telemetry Stream export.

### Message Processing Metrics

_Instrumentation scope: \`org.mule.runtime.flow.statistics\`_

These metrics track message throughput and processing errors at the flow level. They reflect the
core message and response lifecycle of a Mule application.

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Metric</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td><p><code>dispatched.messages</code></p></td><td><p>Counter</p></td><td><p>Total number of messages dispatched by sources for flow processing.</p></td></tr><tr><td><p><code>received.events</code></p></td><td><p>Counter</p></td><td><p>Total events received by flows for processing.</p></td></tr><tr><td><p><code>execution.errors</code></p></td><td><p>Counter</p></td><td><p>Errors encountered during flow processing.</p></td></tr><tr><td><p><code>fatal.errors</code></p></td><td><p>Counter</p></td><td><p>Unrecoverable errors that terminated flow processing.</p></td></tr></tbody></table>

These metrics are useful for understanding application throughput and detecting error rate trends
over time.

### Flow Inventory Metrics

_Instrumentation scope: \`org.mule.runtime.flow.statistics\`_

These metrics provide a snapshot of the application’s flow topology. They report how many flows of
each category are declared in the application configuration and how many are currently active
(started).

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Metric</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td><p><code>declared-trigger-flows</code></p></td><td><p>Counter</p></td><td><p>Number of flows with message sources (HTTP listeners, schedulers, etc.)</p></td></tr><tr><td><p><code>active-trigger-flows</code></p></td><td><p>Counter</p></td><td><p>Trigger flows currently in a started state.</p></td></tr><tr><td><p><code>declared-private-flows</code></p></td><td><p>Counter</p></td><td><p>Number of sub-flows and private flows (no message source).</p></td></tr><tr><td><p><code>active-private-flows</code></p></td><td><p>Counter</p></td><td><p>Private flows currently in a started state.</p></td></tr><tr><td><p><code>declared-apikit-flows</code></p></td><td><p>Counter</p></td><td><p>Number of flows generated by APIKit from an API specification.</p></td></tr><tr><td><p><code>active-apikit-flows</code></p></td><td><p>Counter</p></td><td><p>APIKit flows currently in a started state.</p></td></tr></tbody></table>

### Error Metrics

_Instrumentation scope: \`org.mule.runtime.errors\`_

These metrics break down runtime errors by their Mule error type, providing more granular error
visibility than the aggregate `execution.errors` counter.

<table><colgroup><col> <col> <col> <col></colgroup><thead><tr><th>Metric</th><th>Type</th><th>Attributes</th><th>Description</th></tr></thead><tbody><tr><td><p><code>error-count</code></p></td><td><p>Counter</p></td><td><p><code>error-type</code>, <code>mule.artifact.id</code></p></td><td><p>Count of errors classified by Mule error type</p></td></tr></tbody></table>

The `error-type` attribute contains the Mule error identifier (e.g., `MULE:COMPOSITE_ROUTING`,
`HTTP:NOT_FOUND`, `MULE:SOURCE_RESPONSE_SEND`), enabling per-type trend analysis and alerting.

> [!NOTE] `error-count` and `execution.errors` are tracked independently. `execution.errors` can
> report a higher total because it counts all execution-level errors, including those not classified
> into a specific error type.

### Runtime Alerts Metrics

_Instrumentation scope: \`org.mule.runtime.diagnostics\`_

These metrics expose the Mule runtime built in alerting system, tracking operational conditions that
may require attention.

<table><colgroup><col> <col> <col> <col></colgroup><thead><tr><th>Metric</th><th>Type</th><th>Attributes</th><th>Description</th></tr></thead><tbody><tr><td><p><code>alert-fuseboard</code></p></td><td><p>Counter</p></td><td><p><code>mule.alert.name</code>, <code>mule.alert.context</code>, <code>mule.artifact.id</code></p></td><td><p>Number of runtime alerts triggered</p></td></tr></tbody></table>

Known alert types:

<table><colgroup><col> <col></colgroup><thead><tr><th>Alert Name</th><th>Description</th></tr></thead><tbody><tr><td><p><code>OTEL_EXPORT_QUEUE_FULL</code></p></td><td><p>The OpenTelemetry export queue reached capacity, causing telemetry data to be delayed or dropped</p></td></tr><tr><td><p><code>ASYNC_LOGGER_RINGBUFFER_FULL</code></p></td><td><p>The Log4j async logger ring buffer is full, indicating logging backpressure</p></td></tr></tbody></table>

The `mule.alert.context` attribute provides additional detail about the alert (e.g.,
`Resource: log records, Action: delayed`).

### JVM Metrics

_Instrumentation scope: \`io.opentelemetry.runtime-telemetry\`_

Standard OpenTelemetry JVM instrumentation metrics providing visibility into the underlying Java
Virtual Machine.

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Metric</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td><p><code>jvm.memory.used</code></p></td><td><p>Gauge</p></td><td><p>Current memory usage by pool (heap/non-heap)</p></td></tr><tr><td><p><code>jvm.memory.committed</code></p></td><td><p>Gauge</p></td><td><p>Memory committed by the JVM per pool</p></td></tr><tr><td><p><code>jvm.memory.limit</code></p></td><td><p>Gauge</p></td><td><p>Maximum memory available per pool</p></td></tr><tr><td><p><code>jvm.memory.used_after_last_gc</code></p></td><td><p>Gauge</p></td><td><p>Memory used after the most recent GC</p></td></tr><tr><td><p><code>jvm.gc.duration</code></p></td><td><p>Histogram</p></td><td><p>Garbage collection pause durations by GC name</p></td></tr><tr><td><p><code>jvm.thread.count</code></p></td><td><p>Gauge</p></td><td><p>Active threads by state</p></td></tr><tr><td><p><code>jvm.class.count</code></p></td><td><p>Gauge</p></td><td><p>Currently loaded classes</p></td></tr><tr><td><p><code>jvm.class.loaded</code></p></td><td><p>Counter</p></td><td><p>Total classes loaded since JVM start</p></td></tr><tr><td><p><code>jvm.class.unloaded</code></p></td><td><p>Counter</p></td><td><p>Total classes unloaded since JVM start</p></td></tr><tr><td><p><code>jvm.cpu.count</code></p></td><td><p>Gauge</p></td><td><p>Available processors</p></td></tr><tr><td><p><code>jvm.cpu.recent_utilization</code></p></td><td><p>Gauge</p></td><td><p>Recent CPU utilization (0.0 to 1.0)</p></td></tr><tr><td><p><code>jvm.cpu.time</code></p></td><td><p>Counter</p></td><td><p>Cumulative CPU time consumed</p></td></tr></tbody></table>

### Direct Telemetry Stream Exporter Metrics

_Instrumentation scopes: `io.opentelemetry.exporters.otlp_grpc_span_exporter`,
`io.opentelemetry.exporters.otlp_grpc_log_exporter`,
`io.opentelemetry.exporters.otlp_grpc_metric_exporter`, `io.opentelemetry.sdk.metrics`,
`io.opentelemetry.sdk.trace`, \`io.opentelemetry.sdk.logs\`_

These metrics show the health and throughput of the Direct Telemetry Stream exporters so you can
monitor the export of the telemetry data.

#### Export Throughput

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Metric</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td><p><code>otel.sdk.exporter.span.exported</code></p></td><td><p>Counter</p></td><td><p>Number of spans for which export has finished (successful or failed)</p></td></tr><tr><td><p><code>otel.sdk.exporter.log.exported</code></p></td><td><p>Counter</p></td><td><p>Number of log records for which export has finished.</p></td></tr><tr><td><p><code>otel.sdk.exporter.metric_data_point.exported</code></p></td><td><p>Counter</p></td><td><p>Number of metric data points for which export has finished.</p></td></tr></tbody></table>

#### Export Backpressure

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Metric</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td><p><code>otel.sdk.exporter.span.inflight</code></p></td><td><p>Gauge</p></td><td><p>Spans passed to the exporter but not yet exported.</p></td></tr><tr><td><p><code>otel.sdk.exporter.log.inflight</code></p></td><td><p>Gauge</p></td><td><p>Log records passed to the exporter but not yet exported.</p></td></tr><tr><td><p><code>otel.sdk.exporter.metric_data_point.inflight</code></p></td><td><p>Gauge</p></td><td><p>Metric data points passed to the exporter but not yet exported.</p></td></tr></tbody></table>

#### Export Performance

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Metric</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td><p><code>otel.sdk.exporter.operation.duration</code></p></td><td><p>Histogram</p></td><td><p>Duration of exporting a batch of telemetry records (in seconds)</p></td></tr><tr><td><p><code>otel.sdk.metric_reader.collection.duration</code></p></td><td><p>Histogram</p></td><td><p>Duration of the metric reader’s collect operation (in seconds)</p></td></tr></tbody></table>

#### Batch Processing

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Metric</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td><p><code>processedSpans</code></p></td><td><p>Counter</p></td><td><p>Number of spans processed by the <code>BlockingBatchSpanProcessor</code> (attribute <code>dropped=true</code> if dropped due to high throughput)</p></td></tr><tr><td><p><code>processedLogs</code></p></td><td><p>Counter</p></td><td><p>Number of log records processed by the <code>BlockingBatchLogRecordProcessor</code></p></td></tr><tr><td><p><code>queueSize</code></p></td><td><p>Gauge</p></td><td><p>Number of items currently queued in the batch processor</p></td></tr></tbody></table>

## Direct Telemetry Stream from Mule Runtime

> [!NOTE] This feature is available only for Advanced and Titanium tiers of MuleSoft Anypoint
> Platform.

Starting with Mule runtime 4.11.0, you can export OpenTelemetry traces and logs outside Anypoint
Monitoring. This allows Mule runtime to export its OpenTelemetry data to a customer-managed
OpenTelemetry collector instead of the Anypoint Monitoring OpenTelemetry pipeline.

This solution uses a shared responsibility model. Size your environment to support telemetry export.
Sizing depends on factors such as Mule app and third-party observability platform performance. Test
to discover additional factors.

When using this direct telemetry stream feature, Anypoint Monitoring OpenTelemetry features, such as
Telemetry Exporter, aren’t available.

### Recommended Architecture for Direct Telemetry Stream

Mule runtime supports exporting all its telemetry data using the OpenTelemetry protocol (OTLP) and
binary/protobuf format. You can configure an endpoint for each signal type (traces, logs, and
metrics), and data is sent to that endpoint. However, we strongly recommend using a collector
because it provides the most flexible and feature-rich architecture for a wide range of use cases
and deployment scenarios.

See [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) for more information.

Key reasons for using a collector include:

- Compatibility

  Mule runtime sends telemetry signals to one endpoint (the collector) in one (OTEL binary/protobuf)
  format, ensuring vendor neutrality. Then, the collector manages exporting signals to the selected
  backends. This is especially useful for monitoring solutions like Azure, which can’t ingest OTEL
  signals directly and require a specific exporter, or Grafana, which implements authentication and
  additional metrics via collector components.

  Exporting the same signal to multiple third-party solutions is only possible through a collector
  because Mule runtime allows one endpoint per signal.

  Most third-party monitoring solutions support or provide their own collector implementations.

- Security

  Applications don’t need external Internet access to reach third-party monitoring solutions.

  The collector handles authentication and authorization against third-party monitoring solutions.
  You can use it as a single point for dedicated network policies.

- Performance

  You can optimize collectors for specific goals, scale independently, and distribute load without
  affecting Mule runtime instances. Configuration changes don’t affect Mule runtime instances.

- Observability

  Collectors include multiple built-in metrics and health checks.

- Centralize data processing

  A collector can centralize data enrichment and transformation for Mule runtime telemetry and
  additional non-Mule services.

### Direct Telemetry Stream Configuration

Mule runtime includes predefined configuration templates for exporting traces and logs.

#### Hybrid Standalone Deployment

Configure hybrid instances by using
[Runtime Manager hybrid server properties](https://docs.mulesoft.com/hybrid-standalone/managing-servers#configure-server-properties)
instead of Java system properties.

Additionally, you can configure a limited set of options at the application level by using
[application deployment properties](https://docs.mulesoft.com/hybrid-standalone/deploying-apps-hybrid-standalone#deployment-configuration).
This configuration is limited to a subset of the available settings.

Service Identification

- `mule.openTelemetry.exporter.resource.service.name`

- `mule.openTelemetry.exporter.resource.service.namespace`

Sampling

- `mule.openTelemetry.tracer.exporter.sampler.type`

- `mule.openTelemetry.tracer.exporter.sampler.arg`

mTLS

- `mule.openTelemetry.exporter.tls.certificatesStrategy`

- `mule.openTelemetry.exporter.tls.certFileLocation`

- `mule.openTelemetry.exporter.tls.keyFileLocation`

- `mule.openTelemetry.exporter.tls.caFileLocation`

- `mule.openTelemetry.exporter.tls.certBase64`

- `mule.openTelemetry.exporter.tls.keyBase64`

- `mule.openTelemetry.exporter.tls.caBase64`

#### CloudHub, CloudHub 2.0 and Anypoint Runtime Fabric Deployments

Configure these deployments by using
[Runtime Manager application properties](https://docs.mulesoft.com/mule-runtime/4.12/configuring-properties#setting-properties-values-in-runtime-manager).

### Direct Telemetry Stream for Traces Configuration

Tracing export setup properties:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.enabled</code></p></div></div></td><td><div><div><p><code>false</code></p></div></div></td><td><div><div><p>Set to <code>true</code> to enable the Direct Telemetry Stream tracing exporter.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.endpoint</code></p></div></div></td><td><div><div><p><code>http://localhost:4317</code> for GRPC, <code>http://localhost:4318/v1/traces</code> for HTTP</p></div></div></td><td><div><div><p>Endpoint where the traces are exported to.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.type</code></p></div></div></td><td><div><div><p><code>GRPC</code></p></div></div></td><td><div><div><p>Transport type. Available values: <code>GRPC</code>, <code>HTTP</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.headers</code></p></div></div></td><td><div><div><p>Empty list</p></div></div></td><td><div><div><p>Comma-separated list of headers included in every export call made to the export endpoint. Example: <code>headerName1=headerValue1,headerName2=headerValue2</code>.</p></div></div></td></tr></tbody></table>

Tracing export tuning properties:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.tracer.level</code></p></div></div></td><td><div><div><p><code>MONITORING</code></p></div></div></td><td><div><div><p>Mule runtime tracing level. Available values: <code>OVERVIEW</code>, <code>MONITORING</code>, <code>DEBUG</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.sampler</code></p></div></div></td><td><div><div><p><code>parentbased_traceidratio</code></p></div></div></td><td><div><div><p>Head sampling strategy used. Available values: <code>always_on</code>, <code>always_off</code>, <code>traceidratio</code>, <code>parentbased_always_off</code>, <code>parentbased_traceidratio</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.sampler.arg</code></p></div></div></td><td><div><div><p>0.1</p></div></div></td><td><div><div><p>Argument for head sampling strategy. For example, sampling percentage for the <code>traceidratio</code> strategy.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.timeout</code></p></div></div></td><td><div><div><p>10 Seconds</p></div></div></td><td><div><div><p>Timeout for every export call made to the export endpoint.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.compression</code></p></div></div></td><td><div><div><p><code>none</code></p></div></div></td><td><div><div><p>Compression algorithm used to compress the tracing data to be sent on every export call made to the export endpoint. Available values: <code>none</code>, <code>gzip</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.backoff.maxAttempts</code></p></div></div></td><td><div><div><p>5</p></div></div></td><td><div><div><p>Maximum retry attempts to export a batch of spans when the export endpoint responds with a retryable error.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.backoff.initial</code></p></div></div></td><td><div><div><p>1 second</p></div></div></td><td><div><div><p>Wait time before the first retry attempt.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.backoff.maximum</code></p></div></div></td><td><div><div><p>5 seconds</p></div></div></td><td><div><div><p>Maximum wait time before a retry attempt.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.backoff.multiplier</code></p></div></div></td><td><div><div><p>1.5</p></div></div></td><td><div><div><p>Multiplier applied to the last wait time before a new retry attempt.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.batch.scheduledDelay</code></p></div></div></td><td><div><div><p>5000 milliseconds</p></div></div></td><td><div><div><p>Maximum interval between two consecutive exports.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.batch.queueSize</code></p></div></div></td><td><div><div><p>2048 spans</p></div></div></td><td><div><div><p>Export queue size. When full, non-fitting spans are handled according to the configured backpressure strategy. A <code>WARNING</code> log is triggered. Example: <code>Export queue overflow: 1 spans have been dropped. Total spans dropped since the export started: 2</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.batch.maxSize</code></p></div></div></td><td><div><div><p>512 spans</p></div></div></td><td><div><div><p>Maximum size of batches exported per span.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.tracer.exporter.batch.backPressure.strategy</code></p></div></div></td><td><div><div><p><code>DROP</code></p></div></div></td><td><div><div><p>Strategy for handling full export queues. Available values: <code>BLOCK</code> or <code>DROP</code>.</p></div></div></td></tr></tbody></table>

Minimal configuration example that exports 50% of the traces to an HTTP OpenTelemetry collector
endpoint:

mule.openTelemetry.tracer.exporter.enabled=true mule.openTelemetry.tracer.exporter.type=HTTP
mule.openTelemetry.tracer.exporter.endpoint=https://your-collector.com/api/v1/traces
mule.openTelemetry.tracer.exporter.sampler.arg=0.5

> [!NOTE] The `mule.openTelemetry.tracer.exporter.sampler.arg` represents the percentage of sampling
> where 0.1 is 10% and 1 is 100%.

### Direct Telemetry Stream for Logs Configuration

Logging export setup properties:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.enabled</code></p></div></div></td><td><div><div><p><code>false</code></p></div></div></td><td><div><div><p>Set to <code>true</code> to enable the Direct Telemetry Stream logging exporter.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.endpoint</code></p></div></div></td><td><div><div><p><code>http://localhost:4317</code> for GRPC, <code>http://localhost:4318/v1/logs</code> for HTTP</p></div></div></td><td><div><div><p>Endpoint for exporting logs.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.type</code></p></div></div></td><td><div><div><p><code>GRPC</code></p></div></div></td><td><div><div><p>Transport type. Available values: <code>GRPC</code>, <code>HTTP</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.headers</code></p></div></div></td><td><div><div><p>Empty list</p></div></div></td><td><div><div><p>Comma-separated list of headers included in every export call made to the export endpoint. Example: <code>headerName1=headerValue1,headerName2=headerValue2</code>.</p></div></div></td></tr></tbody></table>

Logging export tuning properties:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.level</code></p></div></div></td><td><div><div><p><code>INFO</code></p></div></div></td><td><div><div><p>Minimum severity level to export. Mule doesn’t send log records below this level to the export endpoint. Available values: <code>TRACE</code>, <code>DEBUG</code>, <code>INFO</code>, <code>WARN</code>, <code>ERROR</code>, and <code>FATAL</code>. Extended OpenTelemetry severity levels, such as <code>INFO2</code> or <code>WARN3</code>, are accepted and normalized to their base level before Mule applies them because Log4j doesn’t support extended levels.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.timeout</code></p></div></div></td><td><div><div><p>10 Seconds</p></div></div></td><td><div><div><p>Timeout for every export call made to the export endpoint.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.compression</code></p></div></div></td><td><div><div><p><code>none</code></p></div></div></td><td><div><div><p>Compression algorithm used to compress the logging data to be sent on every export call made to the export endpoint. Available values: <code>none</code>, <code>gzip</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.backoff.maxAttempts</code></p></div></div></td><td><div><div><p>5</p></div></div></td><td><div><div><p>Maximum retry attempts to export a batch of log records when the export endpoint responds with a retryable error.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.backoff.initial</code></p></div></div></td><td><div><div><p>1 second</p></div></div></td><td><div><div><p>Wait time before the first retry attempt.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.backoff.maximum</code></p></div></div></td><td><div><div><p>5 seconds</p></div></div></td><td><div><div><p>Maximum wait time before a retry attempt.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.backoff.multiplier</code></p></div></div></td><td><div><div><p>1.5</p></div></div></td><td><div><div><p>Multiplier applied to the last wait time before a new retry attempt.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.batch.scheduledDelay</code></p></div></div></td><td><div><div><p>5000 milliseconds</p></div></div></td><td><div><div><p>Maximum interval between two consecutive exports.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.batch.queueSize</code></p></div></div></td><td><div><div><p>2048 log records</p></div></div></td><td><div><div><p>Export queue size. When full, non-fitting log records are handled according to the configured backpressure strategy. A <code>WARNING</code> log is triggered. Example: <code>Export queue overflow: 1 log records have been dropped. Total dropped log records since the export started: 2</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.batch.maxSize</code></p></div></div></td><td><div><div><p>512 log records</p></div></div></td><td><div><div><p>Maximum size of batches exported per log record.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.logging.exporter.batch.backPressure.strategy</code></p></div></div></td><td><div><div><p><code>BLOCK</code></p></div></div></td><td><div><div><p>Strategy for handling full export queues. Available values: <code>BLOCK</code> or <code>DROP</code>.</p></div></div></td></tr></tbody></table>

> [!NOTE] Use `mule.openTelemetry.logging.exporter.level` only as a Direct Telemetry Stream export
> filter. Configure actual logging levels in the Log4j configuration.
>
> For example, to export `DEBUG` logs for a specific Java package, set the package level in your
> Log4j configuration file:
>
> ```xml
> <Loggers>
>     <AsyncLogger name="com.example.myapp" level="DEBUG"/>
>     <AsyncRoot level="INFO"/>
> </Loggers>
> ```
>
> With `mule.openTelemetry.logging.exporter.level=DEBUG`, Mule exports `DEBUG` logs produced by
> `com.example.myapp`. With `mule.openTelemetry.logging.exporter.level=INFO` (default), Log4j still
> generates and processes those `DEBUG` logs, but Mule doesn’t export them to the Direct Telemetry
> Stream endpoint.

Minimal configuration example that sends logs to an HTTP OpenTelemetry collector endpoint:

mule.openTelemetry.logging.exporter.enabled=true mule.openTelemetry.logging.exporter.type=HTTP
mule.openTelemetry.logging.exporter.endpoint=https://your-collector.com/api/v1/logs
mule.put.trace.id.and.span.id.in.mdc=true

> [!NOTE] Setting the `mule.put.trace.id.and.span.id.in.mdc` property to `true` won’t be necessary
> in future releases because Mule runtime will automatically enable it.

### Direct Telemetry Stream for Metrics Configuration

Metrics export setup properties:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.enabled</code></p></div></div></td><td><div><div><p><code>false</code></p></div></div></td><td><div><div><p>Set to <code>true</code> to enable the Direct Telemetry Stream metrics exporter.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.endpoint</code></p></div></div></td><td><div><div><p><code>http://localhost:4317</code> for GRPC, <code>http://localhost:4318/v1/metrics</code> for HTTP</p></div></div></td><td><div><div><p>Endpoint for exporting metrics.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.type</code></p></div></div></td><td><div><div><p><code>GRPC</code></p></div></div></td><td><div><div><p>Transport type. Available values: <code>GRPC</code>, <code>HTTP</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.headers</code></p></div></div></td><td><div><div><p>Empty list</p></div></div></td><td><div><div><p>Comma-separated list of headers included in every export call made to the export endpoint. Example: <code>headerName1=headerValue1,headerName2=headerValue2</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.jvmMetrics.enabled</code></p></div></div></td><td><div><div><p><code>false</code></p></div></div></td><td><div><div><p>Set to <code>true</code> to enable the Mule runtime JVM metrics.</p></div></div></td></tr></tbody></table>

Metrics export tuning properties:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.interval</code></p></div></div></td><td><div><div><p>60 Seconds</p></div></div></td><td><div><div><p>Timeout for every export call made to the export endpoint.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.timeout</code></p></div></div></td><td><div><div><p>10 Seconds</p></div></div></td><td><div><div><p>Timeout for every export call made to the export endpoint.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.compression</code></p></div></div></td><td><div><div><p><code>none</code></p></div></div></td><td><div><div><p>Compression algorithm used to compress metric data sent on every export call made to the export endpoint. Available values: <code>none</code>, <code>gzip</code>.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.backoff.maxAttempts</code></p></div></div></td><td><div><div><p>5</p></div></div></td><td><div><div><p>Maximum retry attempts to export a metrics measurement when the export endpoint responds with a retryable error.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.backoff.initial</code></p></div></div></td><td><div><div><p>1 second</p></div></div></td><td><div><div><p>Wait time before the first retry attempt.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.backoff.maximum</code></p></div></div></td><td><div><div><p>5 seconds</p></div></div></td><td><div><div><p>Maximum wait time before a retry attempt.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.meter.exporter.backoff.multiplier</code></p></div></div></td><td><div><div><p>1.5</p></div></div></td><td><div><div><p>Multiplier applied to the last wait time before a new retry attempt.</p></div></div></td></tr></tbody></table>

### Metrics Temporality

Mule runtime exports all its metrics using cumulative
[temporality](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality). Each
exported data point represents the total accumulated value of a metric since the application
started, not just the delta since the last export.

If an export fails (including after backpressure retries), that export cycle is dropped without
being queued or persisted. Because metrics are cumulative, a dropped export doesn’t result in
permanent data loss. The next successful export carries the full accumulated state, and the backend
service reconstructs the timeline with only a brief gap in resolution.

If the Direct Telemetry Stream exporter is still engaged in a previous metric export cycle when the
next interval fires, the new cycle is skipped and a `"Exporter busy. Dropping metrics."` `INFO` log
message is triggered. To minimize skipped cycles, keep your collector and backend service reachable
with low latency.

## Advanced Features

This section describes advanced features that are available for OpenTelemetry support in Mule
runtime.

### Tracing Sampling

Sampling helps reduce observability costs. Mule runtime supports Head sampling, which makes the
sampling decision as early as possible. In a Mule application, that moment is when a message source
is about to trigger a message processing operation, for example, when an HTTP listener has received
a request.

Head sampling is a good fit for:

- Supporting use cases with limited budgets for observability.

- Accommodating use cases when there isn’t a collector involved (consider early development stages)
  where observability is also useful.

While Head sampling is simple and efficient, it has limitations. It can’t:

- Sample only traces that contain an error.

- Sample traces based on overall latency.

- Sample traces based on specific attributes on one or more spans in a trace, for example, sample
  more traces originating from a newly deployed service.

- Apply different sampling rates to traces based on certain criteria, for example, traces from
  low-volume services versus high-volume services.

Tail sampling supports these advanced cases but is costly and can affect performance. Therefore,
Mule runtime only supports Head sampling. If needed, implement Tail sampling in the OpenTelemetry
collector.

Regarding sampling algorithms, Mule runtime currently supports these standardized OpenTelemetry
algorithms: `AlwaysOn`, `AlwaysOff`, `ParentBasedAlwaysOn`, `ParentBasedAlwaysOff`, `TraceIdRatio`,
and `ParentBasedTraceIdRatio`.

#### AlwaysOn

Samples 100% of traces. Every span is recorded and exported. This option is expensive in production
due to high data volume.

#### AlwaysOff

Samples 0% of traces. No spans are recorded or exported. Use this option to completely disable
tracing.

#### ParentBasedTraceIdRatio

A parent-aware sampler that:

- Inherits the parent’s sampling decision if a parent span exists.

- Uses ratio-based sampling (like `TraceIdRatio`) if no parent span exists (root span).

This is the default and most commonly recommended sampler for production. It balances cost control
through ratio-based sampling while preserving trace continuity across distributed services.

#### TraceIdRatio

Samples a configurable percentage of traces based on the trace ID. The decision is deterministic
based on the trace ID, ensuring all spans within the same trace are consistently sampled or not.

#### ParentBasedAlwaysOn

A parent-aware sampler that:

- Inherits the parent’s sampling decision if a parent span exists.

- Always samples (like `AlwaysOn`) if no parent span exists (root span).

#### ParentBasedAlwaysOff

A parent-aware sampler that:

- Inherits the parent’s sampling decision if a parent span exists.

- Never samples (similar to `AlwaysOff`) if no parent span exists (root span).

Use this option for services that should only participate in traces initiated elsewhere.

### Tracing Levels (Direct Telemetry Stream Only)

Mule runtime tracing is structured around three base tracing levels: `OVERVIEW`, `MONITORING`, and
`DEBUG`. Each generated span has an associated tracing level. Flow spans use the `OVERVIEW` level,
flow component spans use the `MONITORING` level, and internal flow component execution spans
(parameter resolution, connection obtention, among others) use the `DEBUG` level.

In most cases, the default tracing level is the best choice. Use `DEBUG` for troubleshooting.

## Custom Resource Attributes

In OpenTelemetry, a resource represents the entity that produces telemetry data. Resource attributes
are key-value pairs that identify that entity across systems and environments.

Mule runtime supports configuring fixed-value custom resource attributes that are attached to every
exported trace and log batch.

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.exporter.resource.attributes</code></p></div></div></td><td><div><div><p>Empty list</p></div></div></td><td><div><div><p>Comma-separated list of resource attributes included in every span and logs export batch.</p></div></div></td></tr></tbody></table>

### Custom Service Name and Service Namespace

Service attributes identify the source of the telemetry data, Mule applications in this case. Mule
runtime supports setting the `service.name` and `service.namespace` properties to custom values:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.exporter.resource.service.name</code></p></div></div></td><td><div><div><p>Artifact ID</p></div></div></td><td><div><div><p>Service name used by the exporter for traces and logs.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.exporter.resource.service.namespace</code></p></div></div></td><td><div><div><p><code>mule</code></p></div></div></td><td><div><div><p>Service namespace used by the exporter for traces and logs.</p></div></div></td></tr></tbody></table>

Additionally, a `service.instance.id` identifies the instance of the Mule application and is
automatically generated as a unique identifier on each deployment or redeployment.

> [!NOTE]
>
> - For Standalone deployments, setting a custom service namespace applies to all the applications
>   deployed. Setting `service.name` isn’t supported.

### Custom Attributes

Adding custom data to the generated spans and log records can help with:

- Adding business context, such as an order ID.

- Adding user context, such as a user ID.

- Enabling filtering and searching.

- Improving troubleshooting by adding debug or troubleshooting data.

Mule runtime supports adding custom attributes to spans and log records through the
[Tracing Module](https://docs.mulesoft.com/tracing-module/latest/tracing-module-reference). Any logging variable set by
the module is also added as an attribute of the span that represents the `set-logging-variable`
operation and the spans and log records that are generated during the flow’s execution, including
subflows, flow references, and error handlers.

While useful, custom attributes must be handled with care. Include only relevant information. Too
many attributes can affect performance and reach the feature limits:

- Maximum of 128 attributes per span (additional attributes are dropped)

- Maximum length of 16384 bytes per attribute (exceeding bytes are truncated)

### Backpressure Strategy

Backpressure occurs when export queues for traces or logs are full. Backpressure strategies define
how Mule runtime handles this event:

- `DROP` backpressure strategy: When the queue is full, non-fitting spans or logs are dropped. This
  is the default strategy for traces.

- `BLOCK` backpressure strategy: When the queue is full, this strategy blocks the call until space
  is generated at the export queue for non-fitting spans or log records. This is the default
  strategy for logs because it aligns with the Log4j behavior.

Dropping spans or log records results in missing data at the export destination. Blocking threads,
on the other hand, can cause performance issues in Mule runtime. For both conditions, Mule runtime
generates a `WARNING` log. Internal statistics track historical occurrences, and the log messages
include cumulative totals since the export started. An alert is also triggered and is available at
the
[Mule troubleshooting plugin DIAF files](https://docs.mulesoft.com/mule-runtime/4.12/mule-troubleshooting-plugin#understanding-diagnostic-information-analysis-file-diaf).

Example of a `DROP` backpressure warning (application log):

`Export queue overflow: 2 spans have been dropped. Total spans dropped since the export started: 4`

Example of a `BLOCK` backpressure warning (application log):

`Export queue overflow: 2 spans have been delayed. Total spans delayed since the export started: 4`

Example of a Mule diagnostics alert (DIAF report):

\* OTEL\_EXPORT\_QUEUE\_FULL 1 / 3 / 5 / 8 Details:

- Resource: spans, Action: drop (4 times)
- Resource: logs, Action: delay (4 times)

### Retry Logic

When errors occur during export requests, Mule runtime uses an exponential backoff with jitter to
avoid overwhelming the destination until recovery. Only errors marked as retryable, as defined by
OpenTelemetry standards, trigger retry logic. See
[HTTP transport protocol reference](https://opentelemetry.io/docs/specs/otlp/#failures) and
[gRPC transport protocol reference](https://opentelemetry.io/docs/specs/otlp/#retryable-response-codes)
for more information.

### TLS support (Direct Telemetry Stream Only)

Direct telemetry stream supports configuring the export for traces and logs to use mutual TLS
authentication, which can be used to secure the communication with an external collector.

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.exporter.tls.enabled</code></p></div></div></td><td><div><div><p><code>false</code></p></div></div></td><td><div><div><p>Set to <code>true</code> to enable the use of TLS for every export call made to the export endpoint.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.exporter.tls.certificatesStrategy</code></p></div></div></td><td><div><div><p><code>CERTIFICATES_PATH</code></p></div></div></td><td><div><div><p>The strategy used to load the certificates. It can also be set to <code>CERTIFICATES_BASE64</code>.</p></div></div></td></tr></tbody></table>

`CERTIFICATES_PATH` strategy is supported for Standalone and Hybrid Standalone deployments.
Configure the certificates location when using this strategy:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.exporter.tls.certFileLocation</code></p></div></div></td><td><div><div><p>-</p></div></div></td><td><div><div><p>TLS certificate file path.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.exporter.tls.keyFileLocation</code></p></div></div></td><td><div><div><p>-</p></div></div></td><td><div><div><p>TLS certificate public key file path.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.exporter.tls.caFileLocation</code></p></div></div></td><td><div><div><p>-</p></div></div></td><td><div><div><p>TLS certificate authority file path.</p></div></div></td></tr></tbody></table>

`CERTIFICATES_BASE64` strategy is supported for all the deployments. Encode the certificates in
Base64 when using this strategy:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Property</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td><div><div><p><code>mule.openTelemetry.exporter.tls.caBase64</code></p></div></div></td><td><div><div><p>-</p></div></div></td><td><div><div><p>TLS certificate encoded in Base64.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.exporter.tls.certBase64</code></p></div></div></td><td><div><div><p>-</p></div></div></td><td><div><div><p>TLS certificate public key encoded in Base64.</p></div></div></td></tr><tr><td><div><div><p><code>mule.openTelemetry.exporter.tls.keyBase64</code></p></div></div></td><td><div><div><p>-</p></div></div></td><td><div><div><p>TLS certificate authority encoded in Base64.</p></div></div></td></tr></tbody></table>

Example of how to encode a certificate file on Linux:

base64 -w 0 client.crt > client.crt.b64

## Troubleshooting Direct Telemetry Stream

### Logs

Mule runtime logs are the place to check for connection issues, export errors, and delayed or
dropped traces and logs.

- Connection issues log errors.

- Export failures log errors.

- Dropped or delayed exports log warnings.

### Mule Diagnostics

Mule diagnostics provides information about delayed or dropped traces and logs and are useful for
quick checks. Note that diagnostics are temporal and cover only the last 60 minutes.

Use diagnostics as a starting point and complement your analysis by reviewing the logs.

### Recommendations

Use an OpenTelemetry collector configured to report its internal metrics. See
[Collector internal metrics](https://opentelemetry.io/docs/collector/internal-telemetry/) for more
information. Having a collector with that setup can greatly improve the troubleshooting capabilities
while moving the hard limits of your particular setup from the 3rd party solution to your own
infrastructure where full observability is possible.

Keep standard Log4j logging enabled, or at least a variation that continues to log errors, to allow
troubleshooting. Without this, you won’t be able to troubleshoot OpenTelemetry log export failures.

Do a performance test of your application before enabling this feature in a production environment.
While our performance tests did not show huge performance costs, Direct telemetry stream is not a
zero cost feature and therefore must be tested.

Always consider the limits of your third-party solution and configure your exports accordingly. Mule
runtime can’t compensate for poor downstream performance scenarios without compromising its own
stability.

The most common cause of export errors, delays, or drops is a bottleneck in the ingestion pipeline
(the OpenTelemetry Collector and the third-party solution). Make sure these components operate
within acceptable performance limits.

Be aware that tuning changes, such as modifying the queue size, can impact Mule application
performance. Test these changes carefully before applying them to production.

## See Also

- [Mule Troubleshooting Plugin](https://docs.mulesoft.com/mule-runtime/4.12/mule-troubleshooting-plugin)