---
schema_version: "1"
id: 3b61e5fc33784cddf173b6e68caed0cb5d65a0a3d7f89757f8e5db4660f6db40
title: Scalability Benchmarks
collection: mulesoft
version: current
locale: en-us
portal_url: https://docs.salesforce.com/mulesoft/current/en-us/runtime-fabric/2.6/rtf-scale.md
canonical_url: https://docs.mulesoft.com/runtime-fabric/2.6/rtf-scale
source_site: docs.mulesoft.com
content_type: documentation
content_hash: e20d7b830dcc6d07013ce9c4249bb80d7a28b333b6c7cdf5c8ab816ef1df4d3a
description: MuleSoft Documentation Site
release: "2.6"
latest: false
---
---
product: Runtime Fabric
version: 2.6
---

# Scalability Benchmarks

> For the full documentation index, see: https://docs.mulesoft.com/llms.txt

Runtime Fabric is a core Kubernetes (K8s) software and works alike in all K8s environments which
comply with Runtime Fabric hardware and software requirements. The following scale limits are
benchmarked on Amazon Elastic Kubernetes Service (Amazon EKS) cluster.

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Limitation</th><th>Description</th><th>Enforced by Runtime Fabric</th></tr></thead><tbody><tr><td><p>Deployments per Runtime Fabric</p></td><td><p>The maximum number of deployments being managed by Runtime Fabric at any instance should not exceed 8000 for optimal performance.</p></td><td><p>Recommended</p></td></tr><tr><td><p>Horizontal Pod Autoscaling (HPA)</p></td><td><p>With Runtime Fabric on Amazon EKS, occasional traffic failures of approximately 0.09% may occur when scaling replicas from 1 to 8. These errors are specific to the pod scaling window, so you may experience traffic failure during scaling operations.</p></td><td></td></tr></tbody></table>

Runtime Fabric functionality and management of application deployments is not affected by the
following:

- Size of individual applications

- Size of worker nodes

- Number of pods running per work node

Because Runtime Fabric is K8s based and uses cloud-native orchestration principals, it works
according to the specified configuration of deployments.

Review the following benchmarked metrics to understand various management metrics when compared
between a nearly empty cluster and a cluster running 8000 deployments. The cluster configuration
used is Runtime Fabric on AWS EKS cluster:

<table><colgroup><col> <col></colgroup><thead><tr><th>Configuration</th><th>Description</th></tr></thead><tbody><tr><td><p>VPC and node groups</p></td><td><p>Default configuration done from eksctl. 3 subnets (one subnet per node group), NAT Gateway.</p></td></tr><tr><td><p>AWS NLB</p></td><td><p>Equipped via helm chart at nginx creation.</p></td></tr><tr><td><p>Node Auto Scaler</p></td><td><p>Recommended 3 GB (Memory)/200m vCore (req==limit)</p></td></tr><tr><td><p>Workers</p></td><td><p>m5.4x large, worker node group</p></td></tr><tr><td><p>Monitoring</p></td><td><p>Fixed, 1 Node, Prometheus, monitoring node group, m5.2x large</p></td></tr><tr><td><p>Ingress</p></td><td><p>Fixed, 2 Nodes, Nginx, ingress node group, m5.4xlarge</p></td></tr></tbody></table>

The following graphic shows how long the Runtime Fabric agent takes to restart if invoked by the
customers or cluster upgrade:

![Diagram that shows restart latency between near empty cluster and loaded cluster](/mulesoft/current/en-us/runtime-fabric/2.6/_images/rtf-scale-cluster-upgrade.png)

Runtime Fabric takes approximately 190 seconds to create the apps irrespective of deployment volume.
The following table shows how long it takes for a Mule app to start, stop, or delete:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>App Operations</th><th>Near Empty Cluster</th><th>Loaded</th></tr></thead><tbody><tr><td><p>Start Latency</p></td><td><p>~110 seconds</p></td><td><p>158 seconds</p></td></tr><tr><td><p>Stop Latency</p></td><td><p>~10 seconds</p></td><td><p>20 seconds</p></td></tr><tr><td><p>Delete Latency</p></td><td><p>~9 seconds</p></td><td><p>~28 seconds</p></td></tr></tbody></table>

The following table shows how long a node cordon operation takes to restart if there is a bad node:

<table><colgroup><col> <col> <col></colgroup><thead><tr><th>Node Operation</th><th>Empty Cluster</th><th>Loaded Cluster</th></tr></thead><tbody><tr><td><p>Cordon</p></td><td><p>~2 seconds</p></td><td><p>~2 seconds</p></td></tr><tr><td><p>Drain process</p></td><td><p>~4 seconds</p></td><td><p>~35 seconds</p></td></tr></tbody></table>

Aspects to consider:

- Runtime Fabric agent requires more than default memory and CPU to manage clusters with large-scale
  deployments, you can modify such values using the information in the following table. For a
  cluster with 8000 deployments, use at least:

  <table><colgroup><col> <col> <col></colgroup><thead><tr><th>Cluster Recommendations</th><th>Default</th><th>200 Node/8000 App Resource Settings</th></tr></thead><tbody><tr><td><p>Agent Resources</p></td><td><p>Memory 200 mb request, 500 mb limit. CPU 100m request, 1vCore limit.</p></td><td><p>3 GB (Memory)/2vCore (request == limit)</p></td></tr></tbody></table>

- Note that in Helm you need to disable node watcher `nodeWatcherEnabled` explicitly when you
  operate at a large scale. Node watcher is always enabled by default (`nodeWatcherEnabled: true`).

- The performance of large-scale deployment orchestration through Runtime Fabric is inversely
  proportional to the volume of deployments.

- RTFCTL clusters are supported only for certain benchmarks.

- Using Helm enables you to manage any cluster that requires customized resources for Runtime Fabric
  agent due to the size of the cluster.

- When scaling up Runtime Fabric agent resource limit, consider increasing the request accordingly.

- Try not to run a container unbounded by removing the limit completely. For OpenShift operators,
  removing the limit sets the agent to the default limit setting.