--- schema_version: "1" id: 85d9ab6e41c9b71db9c986aa3889859469acc38a960d0e90d2536f90478fe5c5 title: Deploying and Managing Mule Applications Using Anypoint Controller for Kubernetes collection: mulesoft version: current locale: en-us portal_url: https://docs.salesforce.com/mulesoft/current/en-us/runtime-fabric/latest/deploy-mule-app-anypoint-controller.md canonical_url: https://docs.mulesoft.com/runtime-fabric/latest/deploy-mule-app-anypoint-controller source_site: docs.mulesoft.com content_type: documentation content_hash: bb6725a4e5d5a2a1777bbbb4be698afdaa320bc8c6aedd12eaf39d1fdcf05aef description: MuleSoft Documentation Site release: "3.0" latest: true --- --- product: Runtime Fabric version: 3.0 is-latest-version: true --- # Deploying and Managing Mule Applications Using Anypoint Controller for Kubernetes > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Controller is a Kubernetes operator that enables you to deploy and manage Mule applications on Runtime Fabric using Kubernetes tools and Custom Resources (CR). With Anypoint Controller, you can: - Define application configurations declaratively in versioned manifest files and let the controller automatically reconcile them with Anypoint Platform and your cluster. - Use standard Kubernetes tools and practices to deploy Mule applications as Kubernetes Custom Resources, including `kubectl`, Helm, Terraform Kubernetes Provider, and GitOps workflows. - Leverage Kubernetes features such as external secret managers, volume mounts, and RBAC policies to meet enterprise security requirements. This guide shows you how to: - Install and configure Anypoint Controller in your Runtime Fabric cluster. - Deploy Mule applications using declarative Mule application Custom Resources (CRs). - Monitor application status using Kubernetes-native status conditions. - Update, delete, and manage application lifecycles with standard Kubernetes commands. - Integrate with CI/CD and GitOps tools like ArgoCD, FluxCD, and Terraform Kubernetes provider. - Configure Kubernetes-specific settings including volumes, secrets, and node affinity. - Troubleshoot common deployment and configuration issues. ## Scope and Limitations Anypoint Controller works with these deployment targets, runtime versions, and installation methods: - Runtime Fabric Mule application deployments - Mule runtime engine (Mule) versions 4.6 and higher. Note: Support is only available in patch versions released on or after [November 7, 2025](https://docs.mulesoft.com/release-notes/runtime-fabric/runtime-fabric-runtimes-release-notes#november-7-2025). Any patch version released prior to this date does not support Anypoint Controller. - Helm-based Runtime Fabric 3.0.1 installation or upgrade Anypoint Controller has these limitations and does not support: - CloudHub 2.0 Mule application deployments - Scheduler-based Mule applications - Omni Gateway deployments - `rtfctl`\-based Runtime Fabric installation or upgrade > [!IMPORTANT] Anypoint Runtime Fabric Command-Line Tool (rtfctl) supports the backup and restore of > the Mule Application Custom Resource Definition (CRD). However, Mule application CRs are not > supported for backup and restore using Anypoint Controller. > > You must plan your recovery strategy using your source control repository as the single source of > truth for all application configurations. After a Runtime Fabric cluster restore, you must > re-apply your Mule Application CRs from your repository. ## Before You Begin Anypoint Controller is designed for Kubernetes administrators with cluster access and proficiency in Kubernetes tools such as `kubectl`, Helm, and Custom Resources. Before you begin, review these prerequisites and environment setup requirements. ### Installation Prerequisites To use Anypoint Controller, you must either: - Have an existing Runtime Fabric installation using Helm. - [Install Runtime Fabric](https://docs.mulesoft.com/runtime-fabric/latest/install-helm) or [upgrade Runtime Fabric](https://docs.mulesoft.com/runtime-fabric/latest/upgrade-helm) using Helm. ### Management Limitations Understanding the management model is critical before deploying and managing applications with Anypoint Controller: - Applications created via Anypoint Controller can only be managed through the Custom Resource itself. - To manage existing applications using Anypoint Controller, you must redeploy them as a Kubernetes Custom Resource. - All configuration changes must be applied by updating the Custom Resource definition in your deployment tool and applying it to the cluster. - Stop and Restart functionality is not supported for applications managed by Anypoint Controller. - The Anypoint Platform Runtime Manager UI displays these applications in read-only mode. - Attempts to modify or delete these applications via Anypoint Platform APIs result in an error. ## Install and Configure Anypoint Controller To set up Anypoint Controller, complete these steps: 1. [Configure Anypoint Controller in Helm Chart](#configure-anypoint-controller-in-helm-chart) 2. [Define Mule Application CR](#define-muleapplication-custom-resource) 3. [Configure Volumes and Volume Mounts](#configure-volumes-and-volume-mounts) 4. [Create a Connected App](#create-an-anypoint-connected-app) 5. [Configure Anypoint Credentials](#configure-anypoint-credentials) ### Configure Anypoint Controller in Helm Chart To enable Anypoint Controller, configure these Helm chart settings: - `global.crds.install = true` Installs the MuleApplication Custom Resource Definition (CRD) in the cluster. The CRD is cluster-scoped and accessible by all Runtime Fabric instances in the cluster. - `agent.rtfd.anypointController.enabled = true` Enables the Anypoint Controller feature for the specific Runtime Fabric instance. You can configure these settings independently based on your deployment scenario: - Single Runtime Fabric instance: Set both `global.crds.install = true` and `agent.rtfd.anypointController.enabled = true`. - Multiple Runtime Fabric instances in the same cluster: - First instance: Set `global.crds.install = true` to install the CRD. Optionally, set `agent.rtfd.anypointController.enabled = true` if this instance manages applications. - Additional instances: Set only `agent.rtfd.anypointController.enabled = true`. The CRD installed by the first instance is accessible to all instances. For more information, see [Install Multiple Instances of Runtime Fabric](https://docs.mulesoft.com/runtime-fabric/latest/install-multiple-instances). For New Runtime Fabric Installations (Single Instance) 1. In your `values.yaml` file, add this configuration: ```yaml global: crds: install: true agent: rtfd: anypointController: enabled: true ``` 2. Install the Runtime Fabric agent: ```bash helm install runtime-fabric /rtf-agent \ -f values.yaml \ -n ``` For Multiple Runtime Fabric Instances First Runtime Fabric instance (installs the CRD): ```yaml global: crds: install: true agent: rtfd: anypointController: enabled: true # Optional: enable if this instance manages applications ``` Additional Runtime Fabric instances (use existing CRD): ```yaml global: crds: install: false # CRD already installed by first instance agent: rtfd: anypointController: enabled: true # Enable controller feature for this instance ``` For Upgrading Existing Runtime Fabric Installations 1. Update your `values.yaml` file with the appropriate configuration: ```yaml global: crds: install: true # Set to false if CRD is already installed agent: rtfd: anypointController: enabled: true ``` 2. Upgrade the Runtime Fabric agent: ```bash helm upgrade runtime-fabric /rtf-agent \ -f values.yaml \ -n ``` ### Define Mule Application CR Use the Custom Resource Definition (CRD) to define and manage Mule application configurations similar to any Kubernetes resources through Kubernetes APIs. Anypoint Controller watches the Mule application CR state and matches it against the desired configured state. Define your Mule application as a Custom Resource using your preferred deployment tool (such as YAML manifests with `kubectl`, Terraform, Helm charts, or GitOps tools). The Custom Resource specifies deployment configurations such as CPU and memory resources, replica count, runtime version, application artifact details, properties, volume mounts for secrets and configuration files, and Kubernetes-specific settings. After you apply this file to your Kubernetes cluster, for example by using `kubectl apply -f .yaml`, Anypoint Controller deploys the application based on the specifications. For a complete reference of all available MuleApplicationCR fields and configuration options, see [Understanding Custom Resource Definitions](#understanding-custom-resource-definitions). #### Basic MuleApplication Structure YAML File ```yaml apiVersion: rtf.mulesoft.com/v1 kind: MuleApplication metadata: name: my-hello-world-app namespace: my-apps spec: target: my-rtf-target businessGroup: MyOrg environment: Production runtimeVersion: "4.8.0:2e-java8" application: group: my-org artifactId: hello-world version: "1.0.0" packaging: jar replicas: 2 resources: requests: cpu: 500m memory: 1Gi ``` ### Configure Volumes and Volume Mounts Use the `volumes` and `volumeMounts` fields to mount standard Kubernetes volume types into your Mule application containers, including sensitive data (credentials, API keys, certificates), configuration files, and persistent storage: - `volumes` Defines the storage volumes available to the application, such as Kubernetes Secrets, ConfigMaps, persistent volumes, or external secret stores. - `volumeMounts` Specifies where to mount the volumes inside the application container and makes the data accessible at the specified file path. When you specify `volumes` and `volumeMounts` in the Mule application CR: 1. The controller includes these specifications in the Deployment resource. 2. Kubernetes mounts the volumes into the Mule application container at the specified paths. 3. Your Mule application can access the files at runtime. > [!NOTE] Volumes specified in the Mule application CR are mounted to the main Mule application > container, not to sidecar containers. Example: Mount Secrets from Kubernetes Secrets Use Kubernetes native Secrets to store secret data on the Kubernetes cluster and mount them into your Mule application. 1. Create a Secret in your Kubernetes cluster: ```bash kubectl create secret generic my-database-secret \ --from-literal=db.username=admin \ --from-literal=db.password=secret123 \ -n my-apps ``` 2. Mount it in your Mule application CR: ```yaml spec: volumes: - name: database-secret secret: secretName: my-database-secret volumeMounts: - name: database-secret mountPath: /mnt/secrets readOnly: true ``` The secret files are now accessible at: - `/mnt/secrets/db.username` - `/mnt/secrets/db.password` Example: Mount a TLS Keystore from a Secret 1. Create a Secret containing your keystore: ```bash kubectl create secret generic java-keystore \ --from-file=keystore.jks=./path/to/keystore.jks \ --from-literal=keystore.password=changeit \ -n my-apps ``` 2. Mount it in your Mule application CR: ```yaml spec: volumes: - name: tls-keystore secret: secretName: java-keystore volumeMounts: - name: tls-keystore mountPath: /var/tls readOnly: true jvmArgs: - "-Djavax.net.ssl.keyStore=/var/tls/keystore.jks" - "-Djavax.net.ssl.keyStorePassword=changeit" ``` Your application can now access: - `/var/tls/keystore.jks` - `/var/tls/keystore.password` Example: Mount Configuration Files from a ConfigMap 1. Create a ConfigMap with configuration files: ```bash kubectl create configmap app-configs \ --from-file=log4j2.xml \ --from-file=truststore.jks \ -n my-apps ``` 2. Mount the files in your Mule application CR: ```yaml spec: volumes: - name: app-configs configMap: name: app-configs volumeMounts: - name: app-configs mountPath: /opt/mule/conf readOnly: true ``` Example: Use External Secret Manager (AWS Secrets Manager) To integrate with AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault, use the Secrets Store CSI Driver. > [!NOTE] You must install the Secrets Store CSI Driver in your cluster. Refer to your Secrets > Manager’s documentation for installation instructions. 1. Create a SecretProviderClass: ```yaml apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: aws-database-secrets namespace: my-apps spec: provider: aws parameters: objects: | - objectName: "production/database/credentials" objectType: "secretsmanager" jmesPath: - path: username objectAlias: db_username - path: password objectAlias: db_password ``` 2. Reference in Mule application CR: ```yaml spec: volumes: - name: aws-secrets csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: aws-database-secrets volumeMounts: - name: aws-secrets mountPath: /mnt/secrets/aws readOnly: true ``` 3. Access Secrets: The secrets are available at: - `/mnt/secrets/aws/db_username` - `/mnt/secrets/aws/db_password` 4. Reference them in your Mule configuration: ```xml ``` Example: Use Azure Key Vault To integrate with Azure Key Vault, configure a SecretProviderClass and reference it in your Mule application CR. ```yaml apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: azure-key-vault-secrets namespace: my-apps spec: provider: azure parameters: usePodIdentity: "false" useVMManagedIdentity: "true" userAssignedIdentityID: "" keyvaultName: "my-keyvault" objects: | array: - | objectName: database-password objectType: secret objectVersion: "" tenantId: "" --- apiVersion: rtf.mulesoft.com/v1 kind: MuleApplication metadata: name: my-app spec: # ... other fields ... volumes: - name: azure-secrets csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: azure-key-vault-secrets volumeMounts: - name: azure-secrets mountPath: /mnt/secrets/azure readOnly: true ``` Example: Mount a Persistent Volume To mount a persistent volume for storing data that must survive pod restarts, use a PersistentVolumeClaim. ```yaml spec: volumes: - name: app-data persistentVolumeClaim: claimName: mule-app-pvc volumeMounts: - name: app-data mountPath: /mnt/data ``` Your application can read and write files to `/mnt/data`, and the data persists across pod restarts. Example: Mount an Empty Directory (Temporary Storage) To provide temporary storage that is created when a pod starts and deleted when it stops, use an emptyDir volume. ```yaml spec: volumes: - name: temp-storage emptyDir: sizeLimit: 1Gi volumeMounts: - name: temp-storage mountPath: /tmp/processing ``` ### Create a Connected App Anypoint Controller requires a Connected App to authenticate with Anypoint Platform APIs. The Connected App acts on its own behalf using client credentials. 1. Log in to Anypoint Platform. 2. Navigate to **Access Management** > **Connected Apps**. 3. Click **Create app**. 4. Select **App acts on its own behalf (client credentials)**. 5. Enter a name for your Connected App, such as `anypoint-controller`. 6. Under **Scopes**, grant these permissions:
CategoryPermissionDescription

Exchange

Exchange Viewer

View and download application artifacts from Exchange

OpenID

Profile

Access basic profile information

Runtime Manager

Create Applications

Create new application deployments

Runtime Manager

Delete Applications

Delete existing application deployments

Runtime Manager

Read Runtime Fabrics

View Runtime Fabric target information

7. Click **Save**. 8. Copy the **Client ID** and **Client Secret**. You need these values in the next step. ### Configure Anypoint Credentials Create a Kubernetes Secret `anypoint-credentials` containing the Connected App credentials in the namespace where Runtime Fabric is installed. ```bash kubectl create secret generic anypoint-credentials \ --from-literal=ANYPOINT_CLIENT_ID= \ --from-literal=ANYPOINT_CLIENT_SECRET= \ -n ``` Replace these: - ``: The Client ID from your Connected App - ``: The Client Secret from your Connected App - ``: The namespace where Runtime Fabric is installed (for example, `rtf`) Anypoint Controller monitors this secret and automatically reloads credentials when updated, enabling credential rotation without controller restarts. #### Rotate Credentials To rotate Connected App credentials: 1. Create a new Connected App in Anypoint Platform. 2. Update the secret: ```bash kubectl create secret generic anypoint-credentials \ --from-literal=ANYPOINT_CLIENT_ID= \ --from-literal=ANYPOINT_CLIENT_SECRET= \ -n \ --dry-run=client -o yaml | kubectl apply -f - ``` 3. The controller automatically detects the change and reloads credentials. ## Deploy Mule Applications After installing and configuring Anypoint Controller, you can deploy Mule applications by creating Mule application CRs. The controller automatically synchronizes these resources with Anypoint Platform and deploys them to your Runtime Fabric cluster. For detailed information about CRD fields and Kubernetes permissions, see: - [Understanding Custom Resource Definitions](#understanding-custom-resource-definitions) - [Understanding Runtime Fabric Agent Kubernetes Permissions](#understanding-runtime-fabric-agent-kubernetes-permissions). ### Create Your First Application Follow these steps to deploy a Mule application using Anypoint Controller. #### Prerequisites - Anypoint Controller installed and configured - Connected App credentials configured - Mule application artifact published to Anypoint Exchange - Target Runtime Fabric name from Runtime Manager - Business group and environment names Complete these steps to create your first application: 1. [Create the Application Manifest](#create-the-application-manifest) 2. [Apply the Manifest](#apply-the-manifest) 3. [Verify the Deployment](#verify-the-deployment) #### Create the Application Manifest Create a file named `hello-world-app.yaml`: ```yaml apiVersion: rtf.mulesoft.com/v1 kind: MuleApplication metadata: name: hello-world-app namespace: my-apps spec: # Required: Anypoint Platform Configuration (Immutable) target: my-rtf-target businessGroup: MyOrg/Engineering environment: Production # Required: Runtime Configuration runtimeVersion: "4.8.0:2e-java8" # Required: Application Artifact application: group: my-org artifactId: hello-world version: "1.0.0" packaging: jar # Deployment Configuration replicas: 2 resources: requests: cpu: 500m memory: 1Gi limits: cpu: 1000m memory: 2Gi ``` This example shows the minimum required fields. For a complete list of all available configuration fields, see [Understanding Custom Resource Definitions](#understanding-custom-resource-definitions). #### Apply the Manifest 1. Create the namespace and label it with the Runtime Fabric namespace: ```bash kubectl create namespace my-apps kubectl label namespace my-apps rtf.mulesoft.com/agentNamespace= ``` Replace `` with the namespace where your Runtime Fabric is installed (for example, `rtf`). > [!IMPORTANT] The `rtf.mulesoft.com/agentNamespace` label tells the Runtime Fabric agent which > Mule application CRs to manage. Only one Runtime Fabric instance can manage applications in a > given namespace. 2. Apply the application manifest: ```bash kubectl apply -f hello-world-app.yaml ``` 3. Verify the Custom Resource was created: ```bash kubectl get muleapplication hello-world-app -n my-apps ``` When you apply the Mule application CR, Anypoint Controller perform these actions: 1. Detects the new MuleApplication resource. 2. Validates the configuration. 3. Adds a label to the namespace for synchronization tracking (`rtf.mulesoft.com/role: workers`). 4. Creates the application in Anypoint Platform. 5. Deploys the application to the specified Runtime Fabric target. 6. Continuously monitors and reconciles the desired state. #### Verify the Deployment 1. Check the application status: ```bash kubectl get muleapplication hello-world-app -n my-apps ``` Example output: ```console NAME APPLICATION FILE DESIRED READY ANYPOINT ID ANYPOINT VERSION hello-world-app hello-world 2 2 abc123def v456 ``` 2. View detailed status: ```bash kubectl describe muleapplication hello-world-app -n my-apps ``` 3. Check the application in Runtime Manager: Log in to Anypoint Platform and navigate to Runtime Manager. Your application appears in the applications list in read-only mode, with all management actions (buttons) grayed out. ### Deployment Examples Explore different deployment patterns and configurations for common use cases. Deploy with Properties To deploy an application with configuration properties, use the `properties` field: ```yaml apiVersion: rtf.mulesoft.com/v1 kind: MuleApplication metadata: name: api-gateway namespace: my-apps spec: target: production-rtf businessGroup: MyOrg environment: Production runtimeVersion: "4.8.0:2e-java8" application: group: my-org artifactId: api-gateway version: "2.1.0" packaging: jar replicas: 3 # Application properties properties: - name: http.port value: "8081" - name: api.base.url value: "https://api.example.com" - name: api.key value: "prod-key-12345" secure: true # Encrypt this property - name: max.connections value: "100" resources: requests: cpu: 1000m memory: 2Gi limits: cpu: 2000m memory: 4Gi ``` Apply the manifest: ```bash kubectl apply -f api-gateway.yaml ``` Deploy with External Properties To reference properties from Kubernetes Secrets or ConfigMaps, use the `propertiesFrom` field: 1. Create a Secret with credentials: ```bash kubectl create secret generic database-credentials \ --from-literal=db.username=admin \ --from-literal=db.password=secretPass123 \ --from-literal=db.host=database.example.com \ -n my-apps ``` 2. Create a ConfigMap with configuration: ```bash kubectl create configmap app-config \ --from-literal=api.timeout=30000 \ --from-literal=api.retry.count=3 \ -n my-apps ``` 3. Reference them in your application: ```yaml apiVersion: rtf.mulesoft.com/v1 kind: MuleApplication metadata: name: order-service namespace: my-apps spec: target: production-rtf businessGroup: MyOrg environment: Production runtimeVersion: "4.8.0:2e-java8" application: group: my-org artifactId: order-service version: "1.5.0" packaging: jar replicas: 2 # Mount properties from external sources propertiesFrom: - secretRef: name: database-credentials - configMapRef: name: app-config resources: requests: cpu: 500m memory: 1Gi limits: cpu: 1000m memory: 2Gi ``` 4. Apply the manifest: ```bash kubectl apply -f order-service.yaml ``` Deploy with Autoscaling To enable horizontal pod autoscaling based on CPU utilization: ```yaml apiVersion: rtf.mulesoft.com/v1 kind: MuleApplication metadata: name: high-traffic-api namespace: my-apps spec: target: production-rtf businessGroup: MyOrg environment: Production runtimeVersion: "4.8.0:2e-java8" application: group: my-org artifactId: high-traffic-api version: "3.0.0" packaging: jar # Configure autoscaling autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 resources: requests: cpu: 500m memory: 1Gi limits: cpu: 1000m memory: 2Gi ``` The controller creates a HorizontalPodAutoscaler resource that automatically adjusts the replica count based on CPU utilization. Deploy a Clustered Application To deploy a clustered application with persistent object store: ```yaml apiVersion: rtf.mulesoft.com/v1 kind: MuleApplication metadata: name: workflow-engine namespace: my-apps spec: target: production-rtf businessGroup: MyOrg environment: Production runtimeVersion: "4.8.0:2e-java8" application: group: my-org artifactId: workflow-engine version: "2.0.0" packaging: jar replicas: 3 # Enable clustering and persistence clustered: true persistentObjectStore: true enforceDeployingReplicasAcrossNodes: true resources: requests: cpu: 1000m memory: 2Gi limits: cpu: 2000m memory: 4Gi ``` Common Deployment Scenarios Customize your Mule application deployments for specific use cases. **Deploy to a Specific Namespace** You can deploy applications to any Kubernetes namespace. Create the namespace and label it with the Runtime Fabric namespace: ```bash kubectl create namespace production-apps kubectl label namespace production-apps rtf.mulesoft.com/agentNamespace= kubectl apply -f my-app.yaml -n production-apps ``` The controller watches namespaces that are labeled with `rtf.mulesoft.com/agentNamespace: `, where `` is the namespace where the Runtime Fabric core software is installed. The agent running in that namespace manages Mule application CRs in the labeled application namespace. For more information about namespace permissions and flexibility, see [Understanding Runtime Fabric Agent Kubernetes Permissions](#understanding-runtime-fabric-agent-kubernetes-permissions). **Deploy with Custom Logging** Configure package-level logging for troubleshooting: ```yaml spec: scopeLoggingConfigurations: - scope: com.mulesoft logLevel: DEBUG - scope: org.mule.extension.http logLevel: INFO - scope: root logLevel: INFO ``` **Deploy with JVM Arguments** Specify custom JVM arguments for memory tuning or SSL configuration: ```yaml spec: jvmArgs: - "-Xmx2048m" - "-Xms1024m" - "-Djavax.net.ssl.keyStore=/var/tls/keystore.jks" ``` **Deploy with Node Affinity** Control which nodes run your application: ```yaml spec: pod: nodeSelector: environment: production disktype: ssd tolerations: - key: "dedicated" operator: "Equal" value: "mule-apps" effect: "NoSchedule" ``` For more Kubernetes customization options including affinity rules, topology spread constraints, and security context, see [Understanding Custom Resource Definitions](#understanding-custom-resource-definitions). ## Monitor Mule Applications Anypoint Controller reports application state through standard Kubernetes status conditions. These conditions provide real-time feedback on the application’s deployment and operational status. ### View Application Status Check the status of a Mule application: ```bash kubectl get muleapplication hello-world-app -n my-apps ``` Output: ```console NAME APPLICATION FILE DESIRED READY ANYPOINT ID ANYPOINT VERSION hello-world-app hello-world 2 2 abc123def v456 ``` View detailed status: ```bash kubectl get muleapplication hello-world-app -n my-apps -o yaml ``` Or use `describe`: ```bash kubectl describe muleapplication hello-world-app -n my-apps ``` ### Status Conditions The controller reports these status conditions. To report these conditions as ArgoCD resource health, see [Configure MuleApplication Health in ArgoCD](#configure-muleapplication-health-in-argocd). Status Condition: Applied The `Applied` condition indicates whether the desired configuration was successfully synchronized with Anypoint Platform. **Status: True (Success)** ```yaml status: conditions: - type: Applied status: "True" reason: APISuccessResponse lastTransitionTime: "2024-01-15T10:30:00Z" ``` The controller successfully created or updated the application in Anypoint Platform. The application configuration matches the desired state defined in the CR. **Status: False (Failure)** ```yaml status: conditions: - type: Applied status: "False" reason: APIErrorResponse message: "409 Conflict - Application name 'hello-world-app' already exists" lastTransitionTime: "2024-01-15T10:30:00Z" ``` The controller could not apply the configuration to Anypoint Platform. The `message` field contains the error details. Common Failure Reasons: - Application name already exists (either not created via Anypoint Controller, or created via Anypoint Controller with the same name) - Invalid Runtime Fabric target name - Invalid business group or environment name - Insufficient vCore entitlement - Invalid runtime version - Exchange artifact not found - Connected App lacks required permissions - Network connectivity issues to Anypoint Platform Status Condition: Available The `Available` condition indicates whether the application has at least one available replica and serving traffic. **Status: True (Available)** ```yaml status: readyReplicas: 2 conditions: - type: Available status: "True" reason: HaveReadyReplicas message: "Application is ready to serve traffic" lastTransitionTime: "2024-01-15T10:32:00Z" ``` Has at least one available replica and is serving traffic. **Status: False (Not Available)** ```yaml status: readyReplicas: 0 conditions: - type: Available status: "False" reason: HaveReadyReplicas message: "Application is not ready to serve traffic" lastTransitionTime: "2024-01-15T10:32:00Z" ``` The application doesn’t have ready replicas. Check pod status and logs to diagnose the issue. Status Condition: Progressing The `Progressing` condition indicates whether a deployment rollout is currently underway. This tracks the process of updating pods to match the desired specification. **Status: True (Rolling Out)** ```yaml status: conditions: - type: Progressing status: "True" reason: RollingOut message: "Rolling out: 1/2 replicas updated" lastTransitionTime: "2024-01-15T10:35:00Z" ``` A rollout is in progress. This status is `True` when: - New replicas are being created - Old replicas are being terminated - Replicas are waiting to become ready **Status: False (Idle)** ```yaml status: conditions: - type: Progressing status: "False" reason: RolloutComplete message: "All replicas updated and ready" lastTransitionTime: "2024-01-15T10:36:00Z" ``` The deployment has finished. All replicas are updated to the latest version and are ready. Status Condition: Degraded The `Degraded` condition indicates whether the application pods are failing to start or stabilize. This is critical for identifying issues like `CrashLoopBackOff`, `ImagePullBackOff`, or readiness probe failures. **Status: True (Failing)** ```yaml status: conditions: - type: Degraded status: "True" reason: PodFailure message: "Pod hello-world-app-86dcf987-xyz: CrashLoopBackOff: Back-off restarting failed container" lastTransitionTime: "2024-01-15T10:40:00Z" ``` The application is in a degraded state. One or more pods in the newest ReplicaSet are failing. If a new version is failing but the old version is still serving traffic, the message will indicate this: New version rollout blocked: Pod : . Service remains available on previous version. **Status: Missing (Healthy)** If the `Degraded` condition is not present in the status list, the application is healthy (or at least not reporting specific pod failures). Status Condition: DesiredSpecAppliedOnCluster The `DesiredSpecAppliedOnCluster` condition indicates whether the currently deployed application version in the cluster matches the desired specification version in Anypoint Platform. **Status: True (Synchronized)** ```yaml status: anypointVersion: v789 conditions: - type: DesiredSpecAppliedOnCluster status: "True" reason: DesiredSpecAppliedOnCluster lastTransitionTime: "2024-01-15T10:33:00Z" ``` The running application version matches the desired version. Deployment is complete and synchronized. **Status: False (Out of Sync)** ```yaml status: conditions: - type: DesiredSpecAppliedOnCluster status: "False" reason: DesiredSpecAppliedOnCluster lastTransitionTime: "2024-01-15T10:33:00Z" ``` The cluster is still deploying the updated version, or the deployment is in progress. Additional Status Fields The MuleApplication status includes additional fields that provide operational information about the application. **Anypoint Platform Identifiers** Use `anypointUri` to quickly navigate to the application in Runtime Manager. ```yaml status: anypointId: abc123def456 # Application ID in Anypoint Platform anypointUri: https://anypoint.mulesoft.com/... # Direct link to app anypointVersion: v789 # Deployed spec version targetId: target-123 # Runtime Fabric target ID businessGroupId: bg-456 # Business group ID environmentId: env-789 # Environment ID ``` **Replica Status** The number of replicas that are running and ready to serve traffic. ```yaml status: readyReplicas: 2 ``` **Retry Information** Indicates whether a failed operation is retrievable. ```yaml status: retryable: true ``` **Reconciliation Metadata** Used internally by the controller for change detection and reconciliation. ```yaml status: lastPublishedHash: "a1b2c3d4..." # Hash of last published spec lastPublishedAt: "2024-01-15T10:30:00Z" # Last publish timestamp ``` ## Manage Mule Applications Anypoint Controller enables you to manage the complete lifecycle of your Mule applications using standard Kubernetes commands and GitOps workflows. ### Reconciliation Behavior Anypoint Controller continuously reconciles the desired state: - Synchronizes the application state every 30 seconds. - Compares the CR spec hash with the last published hash to detect changes. - When changes are detected, updates Anypoint Platform automatically. - Retries failed operations with exponential backoff. - Refreshes the state every 10 minutes, even when no changes occur. Manage your application lifecycle: - [Update a Mule Application](#update-a-mule-application) - [Delete a Mule Application](#delete-a-mule-application) ### Update a Mule Application 1. Modify your Mule application YAML file: ```yaml spec: replicas: 5 # Changed from 2 to 5 ``` 2. Apply the change: ```bash kubectl apply -f hello-world-app.yaml ``` 3. Monitor the update: ```bash kubectl get muleapplication hello-world-app -n my-apps -w ``` The controller detects the change and initiates the update automatically. > [!NOTE] Some fields like `target`, `businessGroup`, and `environment` are immutable and can’t be > changed after creation. For a complete list of mutable and immutable fields, see > [Understanding Custom Resource Definitions](#understanding-custom-resource-definitions). ### Delete a MuleApplication To delete an application managed by Anypoint Controller, delete the Mule application CR: ```bash kubectl delete muleapplication -n ``` Example: ```bash kubectl delete muleapplication hello-world-app -n my-apps ``` The controller automatically: 1. Deletes the application from Anypoint Platform. 2. Removes all Kubernetes resources (Deployment, Service, ConfigMaps, and Secrets). 3. Removes the finalizer from the CR. 4. Allows Kubernetes to delete the CR. > [!IMPORTANT] Deletion is permanent. Make sure your version control repository contains the > application configuration in case you need to redeploy it later. ## Integrate with CI/CD and GitOps Tools Anypoint Controller works seamlessly with GitOps and Infrastructure-as-Code tools. ### GitOps Workflow 1. Store MuleApplication manifests in a version control repository. 2. Define branch strategies (for example, `dev`, `staging`, `production`). 3. Use pull requests or merge requests for configuration changes. 4. Automate deployment with GitOps tools. Integrate Anypoint Controller with these tools: - [ArgoCD Integration](#argocd-integration) - [Configure MuleApplication Health in ArgoCD](#configure-muleapplication-health-in-argocd) - [FluxCD Integration](#fluxcd-integration) - [Terraform Integration](#terraform-integration) ### ArgoCD Integration 1. Store your manifests in version control: ```console my-mule-apps/ ├── production/ │ ├── hello-world-app.yaml │ └── order-processing-app.yaml └── staging/ ├── hello-world-app.yaml └── order-processing-app.yaml ``` 2. Create an ArgoCD Application: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: mule-apps-production namespace: argocd spec: project: default source: repoURL: https://github.com/myorg/mule-deployments targetRevision: main path: production destination: server: https://kubernetes.default.svc namespace: mule-apps syncPolicy: automated: prune: true # Delete resources removed from version control selfHeal: true # Revert manual changes syncOptions: - CreateNamespace=true ``` ArgoCD continuously monitors the version control repository and automatically deploys changes. ### Configure MuleApplication Health in ArgoCD ArgoCD sync status and resource health are independent. A `MuleApplication` can be `Synced` when its Kubernetes resource matches the manifest in Git, even when Anypoint Controller can’t apply the requested configuration or the application rollout fails. ArgoCD doesn’t interpret `MuleApplication.status.conditions` automatically. To report Anypoint Controller results as ArgoCD resource health, add this custom health check to the ArgoCD instance that manages your `MuleApplication` resources. Follow the [ArgoCD custom health check documentation](https://argo-cd.readthedocs.io/en/stable/operator-manual/health/) and use the configuration method for your ArgoCD installation. Use this resource-customization key: ```console resource.customizations.health.rtf.mulesoft.com_MuleApplication ``` This key follows the ArgoCD naming convention `resource.customizations.health._`. The API version isn’t included, and the `MuleApplication` kind is case-sensitive. Add this configuration: ```yaml data: resource.customizations.health.rtf.mulesoft.com_MuleApplication: | local hs = {} if obj.metadata ~= nil and obj.metadata.deletionTimestamp ~= nil then hs.status = "Progressing" hs.message = "Deletion is waiting for Anypoint Controller cleanup" return hs end if obj.status == nil or obj.status.conditions == nil then hs.status = "Progressing" hs.message = "Waiting for Anypoint Controller to report status" return hs end local applied = nil local available = nil local progressing = nil local degraded = nil local desiredSpecApplied = nil for _, condition in ipairs(obj.status.conditions) do if condition.type == "Applied" then applied = condition elseif condition.type == "Available" then available = condition elseif condition.type == "Progressing" then progressing = condition elseif condition.type == "Degraded" then degraded = condition elseif condition.type == "DesiredSpecAppliedOnCluster" then desiredSpecApplied = condition end end -- A desired configuration that failed to apply is terminally unhealthy -- even if an older version still has ready replicas. if applied ~= nil and applied.status == "False" then hs.status = "Degraded" hs.message = applied.message or applied.reason or "Desired configuration was not applied" return hs end if degraded ~= nil and degraded.status == "True" then hs.status = "Degraded" hs.message = degraded.message or degraded.reason or "MuleApplication workload is degraded" return hs end -- The controller can report a terminal Deployment timeout without adding -- Degraded=True when it cannot derive a pod-level failure. if progressing ~= nil and progressing.status == "False" and progressing.reason == "ProgressDeadlineExceeded" then hs.status = "Degraded" hs.message = progressing.message or "MuleApplication deployment exceeded its progress deadline" return hs end if progressing ~= nil and progressing.status == "True" then hs.status = "Progressing" hs.message = progressing.message or progressing.reason or "MuleApplication rollout is progressing" return hs end if desiredSpecApplied ~= nil and desiredSpecApplied.status == "False" then hs.status = "Progressing" hs.message = desiredSpecApplied.message or desiredSpecApplied.reason or "Waiting for the desired specification to reach the cluster" return hs end if applied ~= nil and applied.status == "True" and available ~= nil and available.status == "True" then hs.status = "Healthy" hs.message = available.message or applied.message or "MuleApplication is applied and available" return hs end hs.status = "Progressing" if progressing ~= nil and progressing.message ~= nil then hs.message = progressing.message elseif progressing ~= nil and progressing.reason ~= nil then hs.message = progressing.reason elseif applied ~= nil and applied.message ~= nil then hs.message = applied.message elseif available ~= nil and available.message ~= nil then hs.message = available.message else hs.message = "Waiting for the current MuleApplication generation to become applied and available" end return hs ``` This custom health check maps these `MuleApplication` conditions:
MuleApplication StateArgoCD HealthDescription

The resource has a deletion timestamp

Progressing

Anypoint Controller cleanup is in progress.

Status or conditions aren’t present

Progressing

Anypoint Controller hasn’t reported status.

Applied=False

Degraded

Anypoint Controller couldn’t apply the requested configuration. ArgoCD displays the controller message when available.

Degraded=True

Degraded

Anypoint Controller found a workload failure, such as a pod startup failure.

Progressing=False with reason ProgressDeadlineExceeded

Degraded

The Kubernetes Deployment exceeded its progress deadline. This condition can occur without Degraded=True. For example, an admission failure can prevent creation of a pod that Anypoint Controller can inspect.

Progressing=True

Progressing

A rollout is active.

DesiredSpecAppliedOnCluster=False

Progressing

The desired application version hasn’t reached the cluster.

Applied=True and Available=True

Healthy

The configuration was accepted and at least one replica is available.

Any unknown or incomplete combination

Progressing

This health check uses a conservative fallback and displays the controller message or reason when available.

> [!NOTE] This customization changes only how ArgoCD interprets resource health. It doesn’t change > ArgoCD sync status, modify the `MuleApplication`, or resolve the underlying deployment failure. An > application whose manifest was applied successfully can be `Synced` and `Degraded` at the same > time. `Applied=True` together with `Available=False` isn’t sufficient to classify an application as failed. This combination occurs while an application is starting. `Progressing=False` isn’t always a failure. Anypoint Controller also sets it with reasons such as `RolloutComplete`, `NoReplicas`, and `DeploymentNotFound`. This custom health check maps only the terminal `ProgressDeadlineExceeded` reason to `Degraded`. This customization applies to every ArgoCD-managed `MuleApplication` in that ArgoCD instance. If this resource-customization key is already configured, don’t replace it until you review the existing script. ArgoCD uses one health script for each API group and kind. This script doesn’t require `resource.customizations.useOpenLibs`. ### FluxCD Integration 1. Create a GitRepository resource: ```yaml apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository metadata: name: mule-apps namespace: flux-system spec: interval: 1m url: https://github.com/myorg/mule-deployments ref: branch: main ``` 2. Create a Kustomization resource: ```yaml apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: mule-apps-production namespace: flux-system spec: interval: 10m sourceRef: kind: GitRepository name: mule-apps path: ./production prune: true targetNamespace: mule-apps ``` FluxCD automatically reconciles the cluster state with the version control. ### Terraform Integration Use Terraform to manage MuleApplication resources as infrastructure as code. 1. Create a Terraform configuration file: ```hcl resource "kubernetes_manifest" "hello_world_app" { manifest = { apiVersion = "rtf.mulesoft.com/v1" kind = "MuleApplication" metadata = { name = "hello-world-app" namespace = "mule-apps" } spec = { target = "production-rtf" businessGroup = "MyOrg/Engineering" environment = "Production" runtimeVersion = "4.8.0:2e-java8" application = { group = "my-org" artifactId = "hello-world" version = var.app_version packaging = "jar" } replicas = var.replica_count resources = { requests = { cpu = "500m" memory = "1Gi" } limits = { cpu = "1000m" memory = "2Gi" } } } } } ``` 2. Apply the Terraform configuration: ```bash terraform init terraform plan terraform apply ``` Terraform manages the MuleApplication lifecycle and tracks state changes in your infrastructure. ## Understanding Runtime Fabric Agent Kubernetes Permissions The Runtime Fabric agent requires specific Kubernetes permissions to manage Mule application CRs: - Namespace Label Requirement To enable the Runtime Fabric agent to manage Mule application CRs in a namespace, you must label the namespace with `rtf.mulesoft.com/agentNamespace=`. Only one Runtime Fabric instance can manage applications in a given namespace. - Namespace Flexibility You can create Mule application CRs in any Kubernetes namespace, as long as the namespace is labeled with `rtf.mulesoft.com/agentNamespace: `. A controller running in a given Runtime Fabric agent pod only watches for Mule application CRs in namespaces that have this matching label. - Resource Creation The controller creates application resources (Deployments, Services, ConfigMaps, Secrets) in the same namespace as the Mule application CR. - Label Management The controller adds labels to resources for tracking, ownership, and reconciliation purposes. ## Understanding Custom Resource Definitions The MuleApplication Custom Resource Definition (CRD) defines the schema for Mule application deployments. When you enable Anypoint Controller, Helm installs this CRD in your cluster. ### View the CRD This command verifies that the Mule application CRD is installed in your cluster. ```bash kubectl get crd muleapplications.rtf.mulesoft.com ``` ### Describe the CRD Schema This command shows all available fields and their descriptions. ```bash kubectl explain muleapplication.spec ``` ### Basic MuleApplication Example Create a file named `hello-world-app.yaml`: ```yaml apiVersion: rtf.mulesoft.com/v1 kind: MuleApplication metadata: name: hello-world-app namespace: my-apps spec: # Required: Anypoint Platform Configuration (Immutable) target: my-rtf-target businessGroup: MyOrg/Engineering environment: Production # Required: Runtime Configuration runtimeVersion: "4.8.0:2e-java8" # Required: Application Artifact application: group: my-org artifactId: hello-world version: "1.0.0" packaging: jar # Deployment Configuration replicas: 2 resources: requests: cpu: 500m memory: 1Gi limits: cpu: 1000m memory: 2Gi ``` Apply the Custom Resource: ```bash kubectl apply -f hello-world-app.yaml ``` Anypoint Controller detects the new resource and: 1. Creates the application in Anypoint Platform. 2. Deploys the application to the specified Runtime Fabric target. 3. Continuously monitors and reconciles the desired state. ### MuleApplication Field Reference For a comprehensive reference of all Mule application deployment fields and configuration options, see [Mule Application Deployment Fields](https://docs.mulesoft.com/runtime-fabric/latest/deploy-to-runtime-fabric#rtf-deploy-reference). ### Advanced Field Configurations Required Fields (Immutable After Creation) These fields are required and can’t be changed after the application is created: - `target` (string): The name of the Runtime Fabric target as shown in Runtime Manager. ```yaml spec: target: production-rtf ``` - `businessGroup` (string): The hierarchical path of the Anypoint business group. Use forward slashes to separate parent and child groups. ```yaml spec: businessGroup: Acme/Finance/MyGroup ``` If your business group is the root organization, use only the organization name: ```yaml spec: businessGroup: Acme ``` - `environment` (string): The name of the Anypoint environment (not the environment ID). ```yaml spec: environment: Production ``` Required: Runtime Version The `runtimeVersion` (string) specifies the Mule runtime version to use. Must be version 4.6 or above. ```yaml spec: runtimeVersion: "4.8.0:2e-java8" ``` Required: Application Artifact The `application` field specifies the artifact to deploy from Anypoint Exchange, using Maven coordinate conventions: ```yaml spec: application: group: my-business-group # Business group that owns the artifact artifactId: order-processing # Artifact name version: "1.0.5" # Artifact version packaging: jar # Artifact format (typically "jar") ``` Deployment Configuration **Replicas** The `replicas` field specifies the number of application instances: ```yaml spec: replicas: 3 ``` - Default: `1` - Maximum: `16` replicas per application **Resource Allocation** The `resources` field defines CPU and memory resources per replica: ```yaml spec: resources: requests: cpu: 500m # Minimum CPU (millicores) memory: 1Gi # Minimum memory limits: cpu: 2000m # Maximum CPU memory: 2Gi # Maximum memory ``` **Update Strategy** The `updateStrategy` field controls how updates are rolled out: ```yaml spec: updateStrategy: rolling # or "recreate" ``` - `rolling` (default): Updates replicas incrementally, maintaining availability - `recreate`: Terminates all replicas before creating new ones **Horizontal Pod Autoscaling** The `autoscaling` field configures CPU-based autoscaling: ```yaml spec: autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 ``` When autoscaling is enabled, the controller creates a HorizontalPodAutoscaler resource that automatically adjusts replica count based on CPU utilization. **Clustering and Persistence** Configure Mule clustering for high availability and enable persistent object store for stateful applications. ```yaml spec: clustered: true # Enable Mule clustering persistentObjectStore: true # Enable persistent object store ``` **Replica Distribution** When enabled, Kubernetes ensures replicas are distributed across different nodes for high availability. ```yaml spec: enforceDeployingReplicasAcrossNodes: true ``` Application Properties Define properties directly in the Custom Resource: ```yaml spec: properties: - name: http.port value: "8081" - name: api.key value: "my-api-key-123" secure: true # Encrypt this property - name: max.connections value: "100" ``` Set `secure: true` for sensitive values that should be encrypted in Anypoint Platform. JVM Configuration Specify custom JVM arguments: ```yaml spec: jvmArgs: - "-Xmx2048m" - "-Xms1024m" - "-Djavax.net.ssl.keyStore=/var/tls/keystore.jks" - "-Djavax.net.ssl.keyStorePassword=changeit" ``` Logging Configuration Configure package-level logging: ```yaml spec: scopeLoggingConfigurations: - scope: com.mulesoft logLevel: DEBUG - scope: org.mule.extension.http logLevel: INFO - scope: org.springframework logLevel: WARN - scope: root logLevel: INFO ``` Valid log levels: `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL` Monitoring and Tracing Control Anypoint Monitoring log forwarding and enable OpenTelemetry tracing for application observability. ```yaml spec: disableAmLogForwarding: false # Keep Anypoint Monitoring log forwarding enabled tracingEnabled: true # Enable OpenTelemetry tracing (requires Mule 4.6+) ``` Kubernetes Customization Customize pod, service, and deployment metadata: ```yaml spec: pod: metadata: labels: team: engineering cost-center: "12345" app-version: "1.0.0" annotations: prometheus.io/scrape: "true" prometheus.io/port: "8081" # Node placement nodeSelector: environment: production disktype: ssd # Tolerations for tainted nodes tolerations: - key: "dedicated" operator: "Equal" value: "mule-apps" effect: "NoSchedule" # Affinity rules affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/arch operator: In values: - amd64 podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app: hello-world-app topologyKey: kubernetes.io/hostname # Topology spread constraints topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: hello-world-app # Security context containerSecurityContext: seccompProfile: type: RuntimeDefault runAsNonRoot: true # Service account automountServiceAccountToken: true # Environment variables (only specific keys are supported) env: ENABLE_CONSOLE_LOG: "true" ENABLE_READINESS_PROBE_ON_HTTP_GET: "true" # Ephemeral storage ephemeralStorageLimit: 4Gi service: metadata: labels: monitoring: enabled annotations: service.beta.kubernetes.io/aws-load-balancer-type: "nlb" deployment: metadata: labels: deployment-version: "v1" annotations: fluxcd.io/automated: "true" ``` > [!IMPORTANT] The `pod.env` field uses `map[string]string` format (for example, > `ENABLE_CONSOLE_LOG: "true"`), not a list of `name`/`value` pairs. As with the > [KubernetesTemplate](https://docs.mulesoft.com/runtime-fabric/latest/customize-kubernetes-crd) CR, it supports only the keys listed in this list. > Any other key (for example, `MY_CUSTOM_VAR`) is ignored. > > - `ENABLE_CONSOLE_LOG`: Enables console logging for the application container. Emitted as a > container environment variable. For more information, see > [Enable Console Logging](https://docs.mulesoft.com/runtime-fabric/latest/customize-kubernetes-crd#enable-console-logging). > > - `ENABLE_READINESS_PROBE_ON_HTTP_GET`: Enables the native Kubernetes HTTP GET probe for > readiness. Requires Mule runtime engine patches from November 7, 2025 or later; older patches > might fail to become ready. Emitted as a container environment variable. For more information, > see [Enable Readiness Probe on HTTP GET](https://docs.mulesoft.com/runtime-fabric/latest/customize-kubernetes-crd#enable-readiness-probe). > > - `DISABLE_PER_APP_SA`: Disables the per-application service account, so the Mule application uses > the default namespace service account. Controls deployment-level behavior. For more information, > see [Disable Per Application Service Account](https://docs.mulesoft.com/runtime-fabric/latest/customize-kubernetes-crd#disable-service-account). > > To pass custom configuration values to your Mule application, use the `properties` or > `propertiesFrom` fields instead. For more information, see > [Mounting Properties from External Sources](#mounting-properties-from-external-sources). ### Mounting Properties from External Sources The `propertiesFrom` field in the Mule application CR `spec` enables you to inject application properties from Kubernetes Secrets and ConfigMaps without hardcoding them in the Custom Resource. ```yaml spec: propertiesFrom: - secretRef: name: my-secret - configMapRef: name: my-config ``` When you reference a Secret or ConfigMap in `propertiesFrom`, the controller: 1. Creates `volume` and `volumeMount` configurations automatically. 2. Mounts the files to `/etc/mule/properties/` in the Mule application container. 3. The Mule agent scans this directory recursively and loads properties before starting the application. #### Supported Property Formats The Mule agent supports two property file formats: - Java Properties Files (`.properties` extension): - Files ending with `.properties` are parsed as standard Java properties files - Each line follows the `key=value` format - File-as-Property (any other file): - The filename becomes the property name - The file content becomes the property value - Useful for single-value secrets Properties are loaded alphabetically by source name. Later properties override earlier ones with the same name. Example: Mount Properties from a Secret 1. Create a Secret: ```bash kubectl create secret generic database-credentials \ --from-literal=db.username=admin \ --from-literal=db.password=secretPass123 \ --from-literal=db.host=database.example.com \ -n my-apps ``` 2. Reference it in your MuleApplication: ```yaml spec: propertiesFrom: - secretRef: name: database-credentials ``` 3. Access properties in your Mule application: ```xml ``` Example: Mount Properties from a ConfigMap 1. Create a ConfigMap: ```bash kubectl create configmap app-config \ --from-literal=api.timeout=30000 \ --from-literal=api.retry.count=3 \ --from-literal=api.base.url=https://api.example.com \ -n my-apps ``` 2. Reference it in your MuleApplication: ```yaml spec: propertiesFrom: - configMapRef: name: app-config ``` Example: Mount a Java Properties File 1. Create a properties file: ```properties # application.properties http.port=8081 http.host=0.0.0.0 api.key=prod-key-12345 max.retry.attempts=5 timeout.seconds=30 ``` 2. Create a ConfigMap from the file: ```bash kubectl create configmap app-properties \ --from-file=application.properties \ -n my-apps ``` 3. Reference it in your MuleApplication: ```yaml spec: propertiesFrom: - configMapRef: name: app-properties ``` All properties from the file are loaded and available as `${http.port}`, `${api.key}`, etc. Example: Combine Multiple Property Sources To combine properties from multiple Secrets and ConfigMaps, list them in the order they should be loaded, where later sources override earlier ones if property names conflict. ```yaml spec: propertiesFrom: - secretRef: name: database-credentials # Loaded first - configMapRef: name: app-config # Loaded second (overrides conflicts) - secretRef: name: api-keys # Loaded third (overrides conflicts) ``` Example: Optional Property Sources Make a property source optional (doesn’t fail if not found): ```yaml spec: propertiesFrom: - configMapRef: name: optional-config optional: true # Don't fail if this ConfigMap doesn't exist ``` ### Template Restrictions When using Anypoint Controller, certain Runtime Fabric templates are not supported.
TemplateSupport StatusAlternative

KubernetesTemplate

Not supported

Define all Kubernetes-specific configurations directly in the Mule application CR fields (spec.pod, spec.service, spec.deployment)

HttpRouteTemplate

Not supported

Manage HTTP routing and ingress configurations separately using standard Kubernetes Ingress resources or your cluster’s ingress controller

Ingress Template

Not supported

Manage ingress configurations separately using standard Kubernetes Ingress resources or your cluster’s ingress controller

Templates were designed for imperative application management through Runtime Manager. With Anypoint Controller’s declarative GitOps approach, all configuration must be explicitly defined in the Custom Resource to maintain: - A single source of truth in version control - Complete visibility of application configuration - Reproducible deployments across environments #### Migrating from Templates If you currently use KubernetesTemplate for configuration, migrate those settings to the Mule application CR fields:
Template FieldMule application CR Field

Pod labels

spec.pod.metadata.labels

Pod annotations

spec.pod.metadata.annotations

Node selectors

spec.pod.nodeSelector

Tolerations

spec.pod.tolerations

Affinity rules

spec.pod.affinity

Topology spread

spec.pod.topologySpreadConstraints

Service annotations

spec.service.metadata.annotations

Deployment labels

spec.deployment.metadata.labels

#### Configuring Application URLs and Ingress For application URLs and ingress, you can define Kubernetes native Ingress resources in your cluster for Mule applications created with Anypoint Controller. Use standard Kubernetes Ingress controllers and resources to manage HTTP routing and external access to your applications, rather than Runtime Fabric ingress templates. For more information about configuring ingress for Runtime Fabric, see [Configure Ingress](https://docs.mulesoft.com/runtime-fabric/latest/configure-ingress). ## Troubleshooting Use these techniques to diagnose and resolve common issues with Anypoint Controller and Mule application deployments. ### Check Controller Logs Anypoint Controller is a feature enabled in the existing `rtfd` container. View the logs to troubleshoot deployment issues: ```bash kubectl logs -l app=rtf-agent -n -c rtfd --tail=100 ``` Follow logs in real-time: ```bash kubectl logs -l app=rtf-agent -n -c rtfd -f ``` Enable Debug Logs: By default, the log level is set to `info`, which does not emit detailed information for diagnosis. To enable debug logs for more detailed troubleshooting, add these environment variables to the `app-config` ConfigMap and restart the agent deployment: ```yaml LOG_LEVEL: DEBUG LOGGING_LEVEL_ROOT: debug ``` ### Common Issues Application Not Reconciling The `Applied` condition remains `False` or changes aren’t reflected. To resolve this issue: 1. Verify the namespace has the correct label: ```bash kubectl get namespace --show-labels ``` The namespace should have `rtf.mulesoft.com/agentNamespace=`. Add the label if missing: ```bash kubectl label namespace rtf.mulesoft.com/agentNamespace= ``` 2. Check the Connected App credentials: ```bash kubectl get secret anypoint-credentials -n -o yaml ``` 3. Verify the Connected App has required permissions in Anypoint Platform. 4. Check [Check Controller Logs](#check-controller-logs) for errors. Application Deployment Fails The `Applied` condition is `True`, but `Available` is `False`. Common causes: - Invalid runtime version - Artifact not found in Exchange - Insufficient resources (CPU/memory) - Image pull errors To resolve this issue: 1. Check pod status: ```bash kubectl get pods -n -l rtf.mulesoft.com/app= ``` 2. View pod logs: ```bash kubectl logs -n ``` 3. Check pod events: ```bash kubectl describe pod -n ``` Invalid Configuration `kubectl apply` fails with validation errors. To resolve this issue: 1. Validate the YAML syntax: ```bash kubectl apply -f app.yaml --dry-run=server ``` 2. Check immutable field violations (target, businessGroup, and environment can’t be changed after creation). 3. Verify required fields are present. Anypoint Platform API Errors The `Applied` condition shows `APIErrorResponse`. To resolve this issue: 1. Read the error message in the status condition. Common errors: - `409 Conflict`: Application name already exists (created outside Anypoint Controller) - `404 Not Found`: Invalid target, business group, or environment name - `403 Forbidden`: Connected App lacks permissions - `400 Bad Request`: Invalid configuration (for example, runtime version) 2. Verify the configuration matches your Anypoint Platform setup. ## See Also - [Deploy Applications to Runtime Fabric Using Runtime Manager](https://docs.mulesoft.com/runtime-fabric/latest/deploy-to-runtime-fabric) - [Install Runtime Fabric Using Helm](https://docs.mulesoft.com/runtime-fabric/latest/install-helm) - [Monitor Applications](https://docs.mulesoft.com/runtime-fabric/latest/manage-monitor-applications) - [Customize Kubernetes Resources Using CRDs](https://docs.mulesoft.com/runtime-fabric/latest/customize-kubernetes-crd) - [Configure Horizontal Pod Autoscaling](https://docs.mulesoft.com/runtime-fabric/latest/configure-horizontal-autoscaling) - [Harden Mule Applications](https://docs.mulesoft.com/runtime-fabric/latest/hardening-apps-deployed-runtime-fabric)