--- schema_version: "1" id: 0c1333252e5e25b622de5d719200afe13c2e96c3a88e1cc42bb4faf07135ab4d title: Runtime Fabric High Availability and Disaster Recovery collection: mulesoft version: current locale: en-us portal_url: https://docs.salesforce.com/mulesoft/current/en-us/runtime-fabric/latest/rtf-ha-dr.md canonical_url: https://docs.mulesoft.com/runtime-fabric/latest/rtf-ha-dr source_site: docs.mulesoft.com content_type: documentation content_hash: 47e2452179f39ab795014b3c42d6fa13cbee7740db67c956af2cf28f0ff8ddcf description: MuleSoft Documentation Site release: "3.0" latest: true --- --- product: Runtime Fabric version: 3.0 is-latest-version: true --- # Runtime Fabric High Availability and Disaster Recovery > For the full documentation index, see: https://docs.mulesoft.com/llms.txt Anypoint Runtime Fabric runs Mule applications on a Kubernetes-based runtime plane that you operate on your own infrastructure or cloud. To keep workloads running through failures and to recover from larger disruptions, Runtime Fabric supports both high availability (HA) and disaster recovery (DR). High availability applies at two levels: - Runtime Fabric core software Runtime Fabric provides HA for its own software so that the runtime plane stays operational through node and zone failures, letting you continue deploying and managing applications. - Mule applications You configure HA for your applications separately by running multiple replicas across availability zones. You achieve disaster recovery (DR) by backing up the cluster and restoring it after a larger disruption. MuleSoft doesn’t back up your data or your cluster configuration; you own cluster backup, restore, and DR. For an overview of HA and DR across all Mule deployment models, see [High Availability and Disaster Recovery](https://docs.mulesoft.com/hosting-home/hadr-guide). ## High Availability Versus Disaster Recovery HA and DR both increase availability, but they address different kinds of failure and differ in scope: - High availability HA keeps a system accessible when an individual component fails. On Runtime Fabric, it handles routine, localized failures—such as a node or zone going down—with no loss of service. - Disaster recovery DR restores a system to an acceptable previous state after a broader disaster, such as flooding, power failures, server failures, or misconfigurations. It handles large-scale disruptions and usually incurs a brief loss of service while the DR plan runs. Two measurable objectives drive your DR plan: - Recovery Time Objective (RTO) The maximum downtime a business tolerates. RTO is the time the system takes to recover after a disruption. - Recovery Point Objective (RPO) The maximum data loss a business accepts after a disaster. RPO drives how often you back up data. ## High Availability in Runtime Fabric Runtime Fabric provides high availability for its own core software so that the runtime plane remains operational through node and zone failures. You achieve high availability for your Mule applications separately by spreading replicas across availability zones and externalizing state. - Multiple replicas Deploy each application with a minimum of two replicas. See [Deploying Mule Apps to Runtime Fabric Using Runtime Manager](https://docs.mulesoft.com/runtime-fabric/latest/deploy-to-runtime-fabric). - Availability zones Runtime Fabric uses topology spread constraints to distribute Mule application replicas evenly across availability zones. CPU-based Horizontal Pod Autoscaling (HPA) adjusts the number of application replicas in response to load, but it doesn’t control where Kubernetes schedules them; the topology spread constraints keep the added replicas balanced across zones. HPA applies to Mule applications, not to the Runtime Fabric core software. See [Configuring Horizontal Pod Autoscaling (HPA) for Runtime Fabric Deployments](https://docs.mulesoft.com/runtime-fabric/latest/configure-horizontal-autoscaling). - Failover When a node or zone fails, Kubernetes reschedules replicas onto healthy nodes to maintain availability. - Externalized state Persist data in object storage across application replicas and restarts, so that no state is lost when a replica reschedules. ## Disaster Recovery in Runtime Fabric Runtime Fabric backs up and restores cluster services and the state that deployed applications and their configurations require. Because you operate the cluster, you own disaster recovery. - Backups Schedule regular backups with `rtfctl`, and store them on external storage outside the cluster. MuleSoft recommends hourly backups. The backup covers cluster configuration—services, secrets, ingress, and ConfigMaps—but it doesn’t cover application state, Ops Center data, or control-plane data. Application deployments aren’t part of the backup; Runtime Fabric reconciles them from the Anypoint Platform control plane on restore. See [Backing Up and Restoring Runtime Fabric](https://docs.mulesoft.com/runtime-fabric/latest/manage-backup-restore). - Restore targets Restore to an existing Runtime Fabric cluster, or to a new Kubernetes cluster that uses the same configuration as the backed-up cluster. - Common scenarios Use backup and restore for major infrastructure upgrades, cluster migrations (from on-premises to cloud, or between cloud providers), and failover to a standby cluster. - Anypoint configuration Back up your own Anypoint configurations—such as VPN routes, private space firewall rules, connections, and queues—with the Anypoint CLI or Anypoint Platform APIs. MuleSoft manages the Anypoint Platform control plane. You operate the Runtime Fabric cluster, its backups, and its DR. ## Considerations and Limitations Account for these factors when you plan backup and restore for Runtime Fabric: - Concurrent instances Don’t run two concurrent instances of the same Runtime Fabric through backup and restore. Running the original and a restored copy at the same time causes deployment conflicts. - Changes after backup Application deployment changes are persisted in the Anypoint Platform control plane, not in the backup. When you restore the fabric, it synchronizes deployments from the control plane, so created and deleted applications are reconciled automatically. Changes that aren’t stored in the control plane—such as updated cluster configurations and agent upgrades—aren’t restored; reapply those manually. - Cluster and tooling consistency Restore to the same cluster that the control plane lists, and use the same `rtfctl` version that you used to create the backup. ## Best Practices - Keep your integrations stateless, and externalize state to an external store. - Run at least two replicas of each application, spread across availability zones. - Schedule regular backups, and store them on external storage. - Test your restore procedures regularly. ## See Also - [High Availability and Disaster Recovery Across Deployment Models](https://docs.mulesoft.com/hosting-home/hadr-guide) - [Backing Up and Restoring Runtime Fabric](https://docs.mulesoft.com/runtime-fabric/latest/manage-backup-restore) - [Configuring Horizontal Pod Autoscaling (HPA) for Runtime Fabric Deployments](https://docs.mulesoft.com/runtime-fabric/latest/configure-horizontal-autoscaling) - [Hardening Runtime Fabric](https://docs.mulesoft.com/runtime-fabric/latest/hardening-runtime-fabric)