Active-Active Failover: Design Patterns and Failure Modes

Failover Has Three Phases, and Each Has Its Own Latency Budget

Treating "failover time" as one number hides where the time actually goes. It's more useful to budget it in three phases:

• Detection — how long until the system knows a region is unhealthy. Bounded by health check interval, failure threshold, and how deep the check actually probes (see below).
• Decision — how long until something acts on that knowledge. Automated systems make this near-instant; anything requiring human judgment adds minutes at best, and human judgment under incident pressure is often wrong in a crisis.
• Propagation — how long until traffic actually stops flowing to the unhealthy region across every client, cache, and intermediate resolver in the path. This is usually the largest and least controllable phase.

A design that cuts failover time meaningfully almost always attacks all three phases together — faster, deeper health checks; fully automated decisions with no human in the critical path; and a propagation mechanism faster than plain DNS. Optimizing only one phase, most commonly detection, and leaving the other two untouched, is why some "we improved health checks" projects deliver disappointing real-world results.

Health Checks: Shallow vs. Deep

A shallow health check — "does the load balancer get a 200 from a lightweight endpoint" — catches process crashes and network-level failures but misses an enormous and common class of partial failure: the process is up, responding to a simple ping, but the database connection pool is exhausted, or a critical downstream dependency is failing, and every real user request is erroring out.

A deep health check exercises a representative path through the system — touching the database, a cache, and at least one critical downstream dependency — and reports unhealthy if that path fails, even though the process itself is technically alive. The trade-off: deep checks are more expensive to run frequently and can produce false positives if a checked dependency has its own transient blip unrelated to the overall system's health. The common resolution is running both: a fast, frequent shallow check for basic liveness, and a deeper check on a slightly longer interval feeding the actual failover decision.

Traffic-Shifting Mechanisms, Fastest to Slowest

• Anycast at the network layer — the same IP address is announced from multiple regions via BGP, and network routing itself directs a client to the nearest healthy point of presence. Failover here can be near-instant because it happens below the application entirely, but it requires infrastructure most teams don't own directly — this is what a global accelerator or CDN provider gives you.
• Load balancer / global accelerator health-aware routing — a managed layer continuously health-checks each region's endpoint and stops routing to an unhealthy one, typically within the health check's own detection window. This is usually the fastest mechanism available to most application teams without owning network-layer infrastructure.
• DNS-based failover — a DNS record's answer changes based on health check state. Simple to reason about, but subject to caching behavior across the internet that the origin doesn't fully control — some resolvers and client libraries cache well past a configured TTL.

Automating the Decision, Not Just the Detection

A health check that flags a region unhealthy is only useful if something acts on it immediately and correctly. Two patterns dominate in mature active-active systems: fully automated failover, where the traffic-shifting layer itself consumes health signals and reroutes without a human in the loop, reserved for well-tested, well-understood failure signatures; and automated failover with a fast override, where the system fails over automatically by default but gives on-call engineers a clear, fast path to intervene if the automation is behaving unexpectedly. What almost never works well under real incident pressure is a fully manual failover process that depends on a human correctly diagnosing a regional failure and executing a runbook within a tight time budget — cognitive load and stress during a real incident reliably make this slower and more error-prone than the runbook assumed when it was written calmly.

Failure Modes That Undermine Active-Active

• Flapping health checks — a region oscillating between healthy and unhealthy due to a transient, self-resolving issue causes repeated failovers, each with its own disruption cost. Requiring a small number of consecutive failures (and successes, before failing back) dampens this at a modest latency cost.
• Cascading failure from failover itself — if the surviving region wasn't provisioned to absorb 100% of traffic, a failover can overload it and turn a partial regional outage into a global one. Active-active only delivers its promise if both regions are genuinely sized for full traffic, not half.
• Stale or incomplete health signals — a health check that only probes one component of a multi-tier system can report healthy while a critical downstream dependency is actually failing. This is the deep-health-check problem above, and it's the most common root cause of "the dashboard said healthy but users saw errors" incidents.
• Failback executed too eagerly — automatically routing traffic back to a region the moment it reports healthy again, without a stabilization window, risks flapping straight back into the same failure if the underlying cause hasn't fully cleared.

Putting the Phases Together

A design that achieves an 85% reduction in failover time relative to a slower baseline typically gets there by compounding smaller wins across all three phases rather than one dramatic change: deep health checks on a tight interval with a small consecutive-failure threshold for fast, accurate detection; fully automated, pre-tested failover logic removing human decision latency; and a health-aware load-balancing or global-accelerator layer for propagation instead of relying on DNS caching behavior outside the system's control. None of these individually is exotic — the result comes from applying all of them together and, critically, testing the whole chain under simulated failure regularly (the subject of Article 9), rather than trusting the design on paper.

Frequently Asked Questions

Why is DNS-only failover usually too slow on its own?
Because DNS TTLs are honored inconsistently by resolvers and clients across the internet — some caches ignore a low TTL entirely. A DNS change can take anywhere from seconds to tens of minutes to fully propagate, which is why production failover designs layer a faster mechanism, typically at the load balancer or Anycast layer, on top of DNS rather than relying on DNS alone.

What is a good default health check interval and threshold?
There's no universal number, but a common, defensible starting point is a check every 5-10 seconds with 2-3 consecutive failures required before marking unhealthy. Faster checks catch failures sooner but increase the risk of a flapping resource being falsely marked down during a transient blip; the right tuning depends on how expensive a false failover is for your system.

Can active-active failover ever be truly zero-downtime?
For a meaningful share of requests, yes — in-flight requests to the failing region and any request that lands during the detection window will still see errors or elevated latency. The realistic goal is minimizing that window and the blast radius within it, not eliminating it. Any claim of literal zero impact should be treated skeptically and verified against real failover drills, not architecture diagrams.