---
schema_version: "1"
id: 2c5b042380a5218c095c18d69dc21e06128d6c33718d30dff6f0795c46a19eaa
title: Scalability Benchmarks
collection: mulesoft
version: current
locale: en-us
portal_url: https://docs.salesforce.com/mulesoft/current/en-us/runtime-fabric/latest/rtf-scale.md
canonical_url: https://docs.mulesoft.com/runtime-fabric/latest/rtf-scale
source_site: docs.mulesoft.com
content_type: documentation
content_hash: bca9bc4f43864d200b24dbc3f56a904501d93244693a38875e8f9a3fc2d6a715
description: MuleSoft Documentation Site
release: "3.0"
latest: true
---
---
product: Runtime Fabric
version: 3.0
is-latest-version: true
---

# Scalability Benchmarks

> For the full documentation index, see: https://docs.mulesoft.com/llms.txt

Runtime Fabric is a core Kubernetes (K8s) software and works alike in all K8s environments which
comply with Runtime Fabric hardware and software requirements.

## Scale Limits

The following scale limits are benchmarked on Amazon Elastic Kubernetes Service (Amazon EKS)
cluster.

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Limitation</th><th>Description</th><th>Enforced by Runtime Fabric</th></tr></thead><tbody><tr><td><p>Deployments per Runtime Fabric</p></td><td><p>The maximum number of deployments being managed by Runtime Fabric at any instance should not exceed 8000 for optimal performance.</p></td><td><p>Recommended</p></td></tr><tr><td><p>Horizontal Pod Autoscaling (HPA)</p></td><td><p>With Runtime Fabric on Amazon EKS, occasional traffic failures of approximately 0.09% may occur when scaling replicas from 1 to 8. These errors are specific to the pod scaling window, so you may experience traffic failure during scaling operations.</p></td><td></td></tr></tbody></table>

Runtime Fabric functionality and management of application deployments is not affected by the
following:

- Size of individual applications

- Size of worker nodes

- Number of pods running per work node

Because Runtime Fabric is K8s based and uses cloud-native orchestration principals, it works
according to the specified configuration of deployments.

## Benchmarked Metrics

Review the following benchmarked metrics to understand various management metrics when compared
between a nearly empty cluster and a cluster running 8000 deployments. The cluster configuration
used is Runtime Fabric on AWS EKS cluster:

<table><colgroup><col> <col></colgroup><thead><tr><th>Configuration</th><th>Description</th></tr></thead><tbody><tr><td><p>VPC and node groups</p></td><td><p>Default configuration done from eksctl. 3 subnets (one subnet per node group), NAT Gateway.</p></td></tr><tr><td><p>AWS NLB</p></td><td><p>Equipped via helm chart at nginx creation.</p></td></tr><tr><td><p>Node Auto Scaler</p></td><td><p>Recommended 3 GB (Memory)/200m vCore (req==limit)</p></td></tr><tr><td><p>Workers</p></td><td><p>m5.4x large, worker node group</p></td></tr><tr><td><p>Monitoring</p></td><td><p>Fixed, 1 Node, Prometheus, monitoring node group, m5.2x large</p></td></tr><tr><td><p>Ingress</p></td><td><p>Fixed, 2 Nodes, Nginx, ingress node group, m5.4xlarge</p></td></tr></tbody></table>

## Runtime Fabric Restart Latency

The following graphic shows how long the Runtime Fabric agent takes to restart if invoked by the
customers or cluster upgrade:

![Diagram that shows restart latency between near empty cluster and loaded cluster](/mulesoft/current/en-us/runtime-fabric/latest/_images/rtf-scale-cluster-upgrade.png)

Runtime Fabric takes approximately 190 seconds to create the apps irrespective of deployment volume.
The following table shows how long it takes for a Mule app to start, stop, or delete:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>App Operations</th><th>Near Empty Cluster</th><th>Loaded</th></tr></thead><tbody><tr><td><p>Start Latency</p></td><td><p>~110 seconds</p></td><td><p>158 seconds</p></td></tr><tr><td><p>Stop Latency</p></td><td><p>~10 seconds</p></td><td><p>20 seconds</p></td></tr><tr><td><p>Delete Latency</p></td><td><p>~9 seconds</p></td><td><p>~28 seconds</p></td></tr></tbody></table>

The following table shows how long a node cordon operation takes to restart if there is a bad node:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Node Operation</th><th>Empty Cluster</th><th>Loaded Cluster</th></tr></thead><tbody><tr><td><p>Cordon</p></td><td><p>~2 seconds</p></td><td><p>~2 seconds</p></td></tr><tr><td><p>Drain process</p></td><td><p>~4 seconds</p></td><td><p>~35 seconds</p></td></tr></tbody></table>

## Considerations

Review the following scale considerations:

- Runtime Fabric agent requires more than default memory and CPU to manage clusters with large-scale
  deployments, you can modify such values using the information in the following table. For a
  cluster with 8000 deployments, use at least:

  <table><colgroup><col> <col> <col></colgroup><thead><tr><th>Cluster Recommendations</th><th>Default</th><th>200 Node/8000 App Resource Settings</th></tr></thead><tbody><tr><td><p>Agent Resources</p></td><td><p>Memory 200Mi request, 500Mi limit. CPU 100m request, 1vCore limit.</p></td><td><p>1000Mi (Memory)/1vCore (request == limit)</p></td></tr><tr><td><p>Daemon Resources</p></td><td><p>Memory 200Mi request, 300Mi limit. CPU 150m request, 200m limit.</p></td><td><p>1200Mi (Memory)/500m (request == limit)</p></td></tr></tbody></table>

- The default resource allocation supports up to 2K apps / 75 nodes.

- The performance of large-scale deployment orchestration through Runtime Fabric is inversely
  proportional to the volume of deployments.

- RTFCTL clusters are supported only for certain benchmarks.

- Using Helm enables you to manage any cluster that requires customized resources for Runtime Fabric
  agent due to the size of the cluster.

- When scaling up Runtime Fabric agent resource limit, consider increasing the request accordingly.

- Try not to run a container unbounded by removing the limit completely. For OpenShift operators,
  removing the limit sets the agent to the default limit setting.

- `resource-cache` and `mule-clusterip-service` services can be scaled horizontally because they are
  stateless.