--- schema_version: "1" id: 0f5feda5cf16f80b3125e996a306ea7e031f7b92f930c18db335814c8ad0936c title: Configuring Horizontal Pod Autoscaling (HPA) for Runtime Fabric Deployments collection: mulesoft version: current locale: en-us portal_url: https://docs.salesforce.com/mulesoft/current/en-us/runtime-fabric/2.8/configure-horizontal-autoscaling.md canonical_url: https://docs.mulesoft.com/runtime-fabric/2.8/configure-horizontal-autoscaling source_site: docs.mulesoft.com content_type: documentation content_hash: c8b0709fd09b45800b995882b84cf8a5fa2355a567a216c3c1ff7925a9a19878 description: MuleSoft Documentation Site release: "2.8" latest: false --- --- product: Runtime Fabric version: 2.8 --- # Configuring Horizontal Pod Autoscaling (HPA) for Runtime Fabric Deployments > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Runtime Fabric instances support horizontal scaling of Mule app deployments by spinning up additional replicas or spinning down unneeded replicas in response to the CPU usage as a signal. Mulesoft supports configuring Mule apps to be responsive to the CPU usage on Kubernetes (K8s) runtime side. You can enable CPU based horizontal autoscaling per application on the Runtime Manager application deployment page, which will result in automatic scale up or scale down of the deployment replicas as needed. In Kubernetes, a Horizontal Pod Autoscaler (HPA) automatically updates a workload resource, with the aim of automatically scaling the workload to match demand. Horizontal scaling means that the response to increased load is to deploy more pods. > [!NOTE] HPA is supported only on Runtime Fabric agent version 2.6.22 and higher. This feature applies only to select customers who are opted into the new pricing and packaging model. For more details, see [Anypoint Platform Pricing](https://docs.mulesoft.com/general/pricing) ## Before You Begin To configure autoscaling, the cluster administrator must enable the [support for metrics APIs](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/) on your managed K8s API server. Refer to your managed vendors documentation on how to install or enable metrics API servers. ## Recommendations for Cluster Autoscaling The following recommendations are for your infrastructure team to configure cluster autoscaling ### Configure Node Groups Configure one node group per availability zone, with a minimum of one node in the node group. This ensures that Kubernetes has information about all the availability zones in the cluster and avoids an issue where zone topology unschedulable pod fails to scale-out topologySpread predicate check. Make sure that nodes from all the node groups have the exact same labels except for the common labels and specific cloud provider labels that are different across nodes or availability zones by default. Custom labels (which are not common labels and specific cloud provider labels) and are different across nodes need to be excluded from cluster autoscaler when evaluating similar node groups. For reference on managed EKS labels such as: eks.amazonaws.com/nodegroup failure-domain.beta.kubernetes.io/region failure-domain.beta.kubernetes.io/zone kubernetes.io/hostname topology.kubernetes.io/region topology.kubernetes.io/zone ### Configure Cluster Autoscaler Flags Start the cluster autoscaler with flag `--balance-similar-node-groups=true` and exclude the custom labels previously mentioned in the node groups section with the flag `--balancing-ignore-label`. These flags ensure that the cluster autoscaler balances the number of nodes across availability zones. For reference on managed EKS cluster, the following node labels are different across node groups: k8s.amazonaws.com/eniConfigCustom vpc.amazonaws.com/eniConfig Therefore, the cluster autoscaler starts with: \- --balancing-ignore-label=k8s.amazonaws.com/eniConfigCustom - --balancing-ignore-label=vpc.amazonaws.com/eniConfig These configuration recommendations serve as a guideline. However, the actual configuration of the cluster autoscaler (CA) depends on the cloud or infrastructure provider (vendor) you use and the relevant topology considerations. You must configure the CA appropriately. ### Configure Overprovisioning Nodes Configure at least one overprovisioning node per availability zone. Each overprovisioning nodes holds a resource that can accommodate three Mule app nodes, which improves the performance when new nodes are needed. > [!NOTE] Refer to your managed vendor documentation for cluster autoscaling best practices and > over-provisioning configuration details. ### Configure Topology Spread Constraints Your Mule app deployments which are enabled with autoscaling, are required to customize the kubernetes resources for the topology spread constraints in pod spec by updating the **whenUnsatisfiable** field with `DoNotSchedule`. This configuration ensures that autoscaling replicas are evenly spread across zones for high availability. topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: rtf.mulesoft.com/id: "{{ .Values.id }}" Using `whenUnsatisfiable: DoNotSchedule` guarantees that pods are evenly spread across availability zones. To modify this setting, use the K8S templates CRD to adjust your deployments using HPA to `whenUnsatisfiable: DoNotSchedule`. You can also use dedicated namespaces (environments) to separate the autoscaling workloads from the classic ones. If you don’t make this change, you use `whenUnsatisfiable: ScheduleAnyway`. For further details on the configuration, refer to [Customizing Mule App Kubernetes Resources](https://docs.mulesoft.com/runtime-fabric/2.8/customize-kubernetes-crd) documentation. > [!WARNING] Customizing topology spread with `whenUnsatisfiable: DoNotSchedule`, without the > recommended configurations for node groups per availability zone, cluster autoscaler, and > overprovisionining, can lead to K8s failing to schedule the replicas with `Pending state` error. ### Understand CPU-based Autoscaling Policy MuleSoft owns and applies the [autoscaling](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/) policy for your Mule application deployments. The following example shows the CPU-based HPA policy used for Mule apps deployed on Runtime Fabric instances. Note that you can choose `minReplicas` and `maxReplicas` values from the Runtime Manager user interface. apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: my-app namespace: app-namespace spec: behavior: scaleDown: policies: - periodSeconds: 15 type: Percent value: 100 selectPolicy: Max stabilizationWindowSeconds: 1800 scaleUp: policies: - periodSeconds: 180 type: Percent value: 100 selectPolicy: Max stabilizationWindowSeconds: 0 maxReplicas: metrics: - resource: name: cpu target: averageUtilization: 70 type: Utilization type: Resource minReplicas: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: my-app Some points to consider: Scale up can occur at most every 180 seconds. Each period, up to 100% of the currently running replicas may be added until the maximum configured replicas is reached. For scaling up there is no stabilization window. When the metrics indicate that the target should be scaled up, the target is scaled up immediately. Scale down can occur at most every 15 seconds. Each Period, up to 100% of the currently running replicas may be removed which means the scaling target can be scaled down to the minimum allowed replicas. The number of replicas removed is based on the aggregated calculations over the past 1800 seconds of the stabilization window. Min replicas - The minimum number of replicas that would be guaranteed to run at any given point of time. - Scale down policy would never remove replicas below this number. Max replicas - The maximum number of replicas that are capped, beyond which no more replicas can be added for scale up. For details, see [limitations-self.adoc#limitations](https://docs.mulesoft.com/runtime-fabric/2.8/limitations-self#limitations). - Scale up policy would never add replicas above this number. Enabling HPA can result in customers incurring additional flow usage when your application scales horizontally. To avoid overages from unpredicted scaling, configure the maximum configured replicas judiciously to stay within purchased flow limits. Track your incurred flow usage through [usage reports](https://docs.mulesoft.com/general/usage-reports). HPA uses CPU utilization at average of 70% HPA bases on [resource requests](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/#how-does-a-horizontalpodautoscaler-work). When target utilization value is set, the controller calculates the utilization value as a percentage of the equivalent resource request on the containers in each pod. If pod average use is greater than 70% in a period (periods for scale-up are set as 180 seconds), HPA scales up. Refer to the Kubernetes documentation for additional details. ## Performance Considerations and Limitations For a successful horizontal autoscaling of your Mule apps, review the following performance considerations: - In Runtime Fabric, the policy in use was benchmarked for Mule apps with CPU Reserved: 0.45vCpu and Limit: 0.55vCpu, which corresponds to these settings: resources: limits: cpu: 550m requests: cpu: 450m - Mule apps that scale based on CPU usage are a good fit with CPU based HPA. For example: - HTTP/HTTPS apps with async requests - Reverse proxies - Low latency + high throughput apps - Dataweave transformations - APIKit Routing - API Gateways with policies - Non-reentrant apps that do not have built in parallel processing such as batch jobs, scheduler apps without re-entrancy and duplicate scheduling across apps and low throughput, high latency apps with large requests may not be a good fit with CPU based HPA. - Scale up and scale down performance can vary based on the replica size and the application type. - The policy is optimized for replica sizes with <850m. Larger replica sizes might take longer to scale. HPA is recommended only for smaller CPU applications with average usage of 0.2vCPU. ### Limitations There are some limitations to consider: - CPU is the only resource for horizontal autoscaling. ## Configure Horizontal Pod Autoscaling To configure horizontal autoscaling for Mule apps deployed to Runtime Fabric, follow these steps: 1. Enable the [support for metrics APIs](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/) on your managed K8s API server. 2. From Anypoint Platform, select **Runtime Manager** > **Applications**. 3. Click **Deploy application**. 4. In the **Runtime** tab, check the **Enable Autoscaling** box. 5. Set the minimum and maximum **Replica Count** limits. 6. Click **Deploy Application**. ![Runtime Manager UI with Enable Autoscaling field selected](/mulesoft/current/en-us/runtime-fabric/2.8/_images/rtf-autoscaling.png) ## Autoscaling Status and Logs When an autoscaling event occurs and your Mule app with horizontal autoscaling scales up, you can check the **Scaling** status in the Runtime Manager UI by clicking **View status** in your application’s details window: ![Runtime Manager UI with Mule app and Scaling status](/mulesoft/current/en-us/runtime-fabric/2.8/_images/rtf-hpa-rtm.png) You can also track when autoscaling events occurred through **Audit logs** in Access Management. Each time an application deployment scaled, there is an audit log published under the product **Runtime Manager**, by the **Anypoint Staff** user. The log has **Action** set to `Scaling` with the **Object** as the application ID. ![Runtime Manager UI with Audit logs and Scaling status](/mulesoft/current/en-us/runtime-fabric/2.8/_images/rtf-hpa-rtm-2.png) The following is an example log payload: ```console {"properties":{"organizationId":"my-orgID-abc","environmentId":"my-envID-xyz","response":{"message":{"message":"Application id:my-appID-123 scaled DOWN from 3 to 2 replicas.","logLevel":"INFO","context":{"logger":"Runtime Manager"},"timestamp":1700234556678}},"deploymentId":"my-appID-123","initialRequest":"/organizations/my-orgID-abc/environments/my-envID-xyz/deployments/my-appID-123/specs/my-specID-456"},"subaction":"Scaling"} ``` Additionally, you can track the scaled up replicas startup and the number of replicas of your Mule apps by running the following `kubectl` command in your terminal: ```console kubectl get events | grep HorizontalPodAutoscaler ``` Use the `kubectl get events` command in Kubernetes to retrieve events that occurred within the cluster. The command provides information about various activities and changes happening in the cluster, such as pod creations, deletions, and other important events. The `grep` command filters the output of `kubectl get events` for lines that contain the string `HorizontalPodAutoscaler`. The following example shows the command output that includes events related to a `HorizontalPodAutoscaler` with information about successful rescaling operations triggered by the HPA: ```console # kubectl get events | grep HorizontalPodAutoscaler 5m20s Normal SuccessfulRescale HorizontalPodAutoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target 5m5s Normal SuccessfulRescale HorizontalPodAutoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target 4m50s Normal SuccessfulRescale HorizontalPodAutoscaler New size: 10; reason: ```