---
schema_version: "1"
id: 2bf150d544894351303a2e96830aada88bd6341b7b6cf3e98cf8710487fe6801
title: Scalability Benchmarks
collection: mulesoft
version: current
locale: en-us
portal_url: https://docs.salesforce.com/mulesoft/current/en-us/runtime-fabric/2.11/rtf-scale.md
canonical_url: https://docs.mulesoft.com/runtime-fabric/2.11/rtf-scale
source_site: docs.mulesoft.com
content_type: documentation
content_hash: c8cce85b519f921a9713ea2b98349244e3aa5ee64b5b37562186e76fa42a54ee
description: MuleSoft Documentation Site
release: "2.11"
latest: false
---
---
product: Runtime Fabric
version: 2.11
---

# Scalability Benchmarks

> For the full documentation index, see: https://docs.mulesoft.com/llms.txt

Runtime Fabric is a core Kubernetes (K8s) software and works alike in all K8s environments which
comply with Runtime Fabric hardware and software requirements.

## Scale Limits

The following scale limits are benchmarked on Amazon Elastic Kubernetes Service (Amazon EKS)
cluster.

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Limitation</th><th>Description</th><th>Enforced by Runtime Fabric</th></tr></thead><tbody><tr><td><p>Deployments per Runtime Fabric</p></td><td><p>The maximum number of deployments being managed by Runtime Fabric at any instance should not exceed 8000 for optimal performance.</p></td><td><p>Recommended</p></td></tr><tr><td><p>Horizontal Pod Autoscaling (HPA)</p></td><td><p>With Runtime Fabric on Amazon EKS, occasional traffic failures of approximately 0.09% may occur when scaling replicas from 1 to 8. These errors are specific to the pod scaling window, so you may experience traffic failure during scaling operations.</p></td><td></td></tr></tbody></table>

Runtime Fabric functionality and management of application deployments is not affected by this:

- Size of individual applications

- Size of worker nodes

- Number of pods running per work node

Because Runtime Fabric is K8s based and uses cloud-native orchestration principals, it works
according to the specified configuration of deployments.

## Benchmarked Metrics

Review these benchmarked metrics to understand various management metrics when compared between a
nearly empty cluster and a cluster running 8000 deployments. The cluster configuration used is
Runtime Fabric on AWS EKS cluster:

<table><colgroup><col> <col></colgroup><thead><tr><th>Configuration</th><th>Description</th></tr></thead><tbody><tr><td><p>VPC and node groups</p></td><td><p>Default configuration done from eksctl. 3 subnets (one subnet per node group), NAT Gateway.</p></td></tr><tr><td><p>AWS NLB</p></td><td><p>Equipped via helm chart at nginx creation.</p></td></tr><tr><td><p>Node Auto Scaler</p></td><td><p>Recommended 3 GB (Memory)/200m vCore (req==limit)</p></td></tr><tr><td><p>Workers</p></td><td><p>m5.4x large, worker node group</p></td></tr><tr><td><p>Monitoring</p></td><td><p>Fixed, 1 Node, Prometheus, monitoring node group, m5.2x large</p></td></tr><tr><td><p>Ingress</p></td><td><p>Fixed, 2 Nodes, Nginx, ingress node group, m5.4xlarge</p></td></tr></tbody></table>

## Runtime Fabric Restart Latency

The following graphic shows how long the Runtime Fabric agent takes to restart if invoked by the
customers or cluster upgrade:

![Diagram that shows restart latency between near empty cluster and loaded cluster](/mulesoft/current/en-us/runtime-fabric/2.11/_images/rtf-scale-cluster-upgrade.png)

Runtime Fabric takes approximately 190 seconds to create the apps irrespective of deployment volume.
The following table shows how long it takes for a Mule app to start, stop, or delete:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>App Operations</th><th>Near Empty Cluster</th><th>Loaded</th></tr></thead><tbody><tr><td><p>Start Latency</p></td><td><p>~110 seconds</p></td><td><p>158 seconds</p></td></tr><tr><td><p>Stop Latency</p></td><td><p>~10 seconds</p></td><td><p>20 seconds</p></td></tr><tr><td><p>Delete Latency</p></td><td><p>~9 seconds</p></td><td><p>~28 seconds</p></td></tr></tbody></table>

The following table shows how long a node cordon operation takes to restart if there is a bad node:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Node Operation</th><th>Empty Cluster</th><th>Loaded Cluster</th></tr></thead><tbody><tr><td><p>Cordon</p></td><td><p>~2 seconds</p></td><td><p>~2 seconds</p></td></tr><tr><td><p>Drain process</p></td><td><p>~4 seconds</p></td><td><p>~35 seconds</p></td></tr></tbody></table>

## Considerations

Review these scale considerations:

- Runtime Fabric agent requires more than default memory and CPU to manage clusters with large-scale
  deployments, you can modify such values using the information in this table. For a cluster with
  8000 deployments, use at least:

  <table><colgroup><col> <col> <col></colgroup><thead><tr><th>Cluster Recommendations</th><th>Default</th><th>200 Node/8000 App Resource Settings</th></tr></thead><tbody><tr><td><p>Agent Resources</p></td><td><p>Memory 200Mi request, 500Mi limit. CPU 100m request, 1vCore limit.</p></td><td><p>1000Mi (Memory)/1vCore (request == limit)</p></td></tr><tr><td><p>Daemon Resources</p></td><td><p>Memory 200Mi request, 300Mi limit. CPU 150m request, 200m limit.</p></td><td><p>1200Mi (Memory)/500m (request == limit)</p></td></tr></tbody></table>

- The default resource allocation supports up to 2K apps / 75 nodes.

- The performance of large-scale deployment orchestration through Runtime Fabric is inversely
  proportional to the volume of deployments.

- RTFCTL clusters are supported only for certain benchmarks.

- Using Helm enables you to manage any cluster that requires customized resources for Runtime Fabric
  agent due to the size of the cluster.

- When scaling up Runtime Fabric agent resource limit, consider increasing the request accordingly.

- Try not to run a container unbounded by removing the limit completely. For OpenShift operators,
  removing the limit sets the agent to the default limit setting.

- `resource-cache` and `mule-clusterip-service` services can be scaled horizontally because they are
  stateless.