Multi-datacenter Kubernetes HA in Switzerland: how it actually works

Replicated control plane, etcd quorum, Geneva-Gland-Lucerne storage: what actually happens when a datacenter goes down.

A Kubernetes cluster in a single-zone configuration generally reaches 99.9% availability, that is up to 8.7 hours of downtime a year. Moving to a multi-datacenter architecture with a replicated control plane raises that to 99.99%, under 53 minutes a year. For a business-critical application, that is the difference between a manageable incident and an interruption that costs.

What most articles on Kubernetes HA leave out: availability is not decided at the worker node level alone. It is won or lost at the control plane and in etcd. Understanding those two layers before configuring anything is the precondition for real high availability.

What really determines a Kubernetes cluster's availability

Two metrics drive every HA architecture decision.

The RTO (Recovery Time Objective) measures the time needed for workloads to be rescheduled onto the healthy sites after a failure. On a single-master cluster that takes several minutes: the failure has to be detected, the control plane rebuilt, then the pods rescheduled. On a multi-master cluster with etcd in quorum, failover is automatic and counted in seconds: the control plane keeps working with no human intervention.

The RPO (Recovery Point Objective) defines how much data may be lost in an incident. It depends directly on the replication strategy of the persistent storage. Synchronous replication targets an RPO (synchronous replication; not a published commitment): no data is lost, because every write is confirmed on all sites before being acknowledged. Asynchronous replication tolerates slight data loss in exchange for lower write latency.

* Hikube managed Kubernetes SLA confirmed on hikube.cloud

These metrics define the objectives. The control plane architecture determines whether they are reachable.

The Hikube control plane architecture across three Swiss sites

The Hikube control plane runs with 3 replicas operated by Hikube, distributed across the three Swiss datacenters: Geneva, Gland and Lucerne. Each replica comprises a kube-apiserver, a kube-scheduler and a kube-controller-manager. An etcd cluster replicated across the three sites completes the set.

The key mechanism is the etcd quorum. For the control plane to keep working, a majority of etcd members must stay available. With 3 members, that quorum is 2. If the Gland datacenter fails completely, Geneva and Lucerne maintain the quorum and the control plane keeps scheduling workloads without interruption. If 2 datacenters fail simultaneously, the quorum is lost and the control plane goes read-only until a third member is restored. That case, two simultaneous site failures, corresponds to a probability below what a 99.99% SLA covers.

This mechanism is entirely operated by Hikube. The customer does not manage etcd, does not watch the quorum, does not trigger failover. The control plane configuration in a Hikube cluster manifest:

controlPlane:
replicas, 3
apiServer:
resourcesPreset, small
controllerManager:
resourcesPreset, small
scheduler:
resourcesPreset, micro

A control plane surviving a site failure is not on its own enough to keep the application available. Pod distribution (topology spread, PDB) is the second lever.

Pods: what Kubernetes does not spread on its own

The worker nodes (node groups) run in the customer tenant. Hikube distributes them automatically across Geneva, Gland and Lucerne: you do not choose the DC. Kubernetes does not spread pods on its own: a Deployment with 3 replicas can put all three in the same datacenter. If that datacenter goes down, the application is unavailable even though the control plane survives on the other two sites.

Two mechanisms correct that behaviour. The first is the topology spread constraint, which forces pods to be distributed between availability zones:

topologySpreadConstraints:
- maxSkew, 1
topologyKey, topology.kubernetes.io/zone
whenUnsatisfiable, DoNotSchedule
labelSelector:
matchLabels:
app, my-application

The second is the PodDisruptionBudget, which guarantees that a minimum number of pods stays available during a rolling update or a maintenance operation:

apiVersion, policy/v1
kind, PodDisruptionBudget
metadata:
name, my-application-pdb
spec:
minAvailable, 2
selector:
matchLabels:
app, my-application

Both configurations are the customer's responsibility, not Hikube's. A cluster correctly configured on the Hikube side but without these constraints on the workload side will not reach the 99.99% SLA in practice. To go further into CI/CD deployment strategies on Kubernetes, our dedicated article covers GitOps pipelines with FluxCD.

StorageClass: the choice that determines your RPO

Hikube offers replication of persistent volumes across the three sites for Kubernetes persistent volumes, each with a different trade-off between availability, performance and operational complexity.

* Source, Hikube documentation, docs.hikube.cloud/services/kubernetes/faq

The decision rule is direct, if data loss is unacceptable, use synchronous replication. If write performance comes first and slight loss is tolerable, asynchronous replication is the right compromise. Hikube block volumes are always geo-redundant (Geneva, Gland, Lucerne), synchronous or asynchronous. The storageClass is defined at cluster manifest level and applies to every PVC created in that cluster. For GPU workloads on Kubernetes, the synchronous replication storageClass is recommended to protect model checkpoints.

Rolling updates without interruption: what Hikube handles, what you handle

Kubernetes version updates are done as rolling updates with no control plane interruption. Hikube rotates the control plane components one by one, maintains the etcd quorum throughout the upgrade, and manages the supported versions. The customer triggers the update by modifying the cluster manifest.

Two constraints apply on the customer side. First, upgrades must be incremental: upgrades stay incremental, minor versions are not skipped. Second, workloads must be configured to survive the rolling update of the worker nodes. A critical Deployment without a PodDisruptionBudget or maxUnavailable: 0 can see its pods interrupted while the nodes rotate.

Recommended configuration for critical workloads during an upgrade:

strategy:
type, RollingUpdate
rollingUpdate:
maxUnavailable, 0
maxSurge, 1

What happens when the Gland datacenter goes down at 3 a.m.

A concrete scenario on a correctly configured Hikube production cluster, 3 control plane replicas across Geneva, Gland and Lucerne, synchronous replication storageClass, nodeGroups with topology spread constraints enabled.

T+0: complete failure of the Gland datacenter. Every component hosted on that site becomes unavailable.

T+15 seconds: the kube-apiserver detects the loss of the Gland etcd member. The quorum holds with Geneva and Lucerne (2 members of 3). The control plane keeps working for reads and writes with no interruption.

T+30 seconds: the kube-controller-manager identifies the orphaned pods previously hosted on Gland. The kube-scheduler reschedules them on the worker nodes available in Geneva and Lucerne, respecting the topology spread constraints.

T+45 to 60 seconds: workloads with minReplicas ≥ 2 and PodDisruptionBudgets configured are fully operational again. Persistent volumes on the synchronous replication storageClass are reachable from Geneva and Lucerne with no data loss.

RTO: an order of magnitude, not a published commitment for correctly configured workloads. RPO targeted (synchronous replication; not a published commitment) with the synchronous replication storageClass. This scenario assumes a configuration following the good practice described in this article. A cluster without topology spread constraints and PodDisruptionBudgets will see a significantly longer RTO.

Summary

Three layers determine the real HA of a Kubernetes cluster: the multi-replica control plane with etcd in quorum across three Swiss sites (operated by Hikube), workload distribution through topology spread constraints and PodDisruptionBudgets (configured by the customer), and the choice of storageClass matched to the target RPO. HA is not switched on with a click. It is a combination of architecture decisions at each layer.

Your Kubernetes cluster has to survive the failure of a Swiss datacenter. See our managed Kubernetes offering →