Eleven milliseconds is long enough to lose quorum

The transfer switch did its job

When utility power failed, the automatic transfer switch kicked to generator backup in 11 milliseconds. That’s within spec. Better than spec, actually. Most vendors guarantee 15ms.

The cluster had opinions about that.

[2024-02-26T03:47:12.389Z] WARNING: Node node-03 lost quorum
[2024-02-26T03:47:12.391Z] ERROR: Raft election triggered
[2024-02-26T03:47:12.504Z] CRITICAL: Minority partition detected on node-03

Eleven milliseconds is enough time for a distributed system to decide the world ended.

---

The physics nobody considered

We’re running a 5-node etcd cluster across two availability zones. The consensus protocol assumes all nodes can talk within 10ms. Our power transfer introduced an 11ms gap, which triggered:

1. Heartbeat timeout: Node-03 missed three heartbeats during transfer
2. Election trigger: Remaining nodes started leader election
3. Split brain: Node-03 came back online and thought it was still leader
4. Write conflict: Two leaders, one cluster, zero good outcomes

The math that matters

Heartbeat interval: 100ms
Election timeout: 150-300ms
Power transfer: 11ms

Missed heartbeats during transfer: ceil(11/100) = 0 (but jitter makes it 3)
Actual result: Election triggered, quorum lost, writes rejected for 2.3 seconds

---

What we changed

We didn’t change the power infrastructure—we can’t make the transfer switch faster than physics allows. Instead:

1. Tuned etcd timeouts: Increased heartbeat interval to 500ms
2. Added jitter tolerance: Election timeout now ranges 2-5 seconds
3. Cross-zone latency budgeting: Built power transfer into our SLA calculations
4. Simulation testing: Chaos engineering with intentional power failures

The transfer switch did exactly what it was rated to do. The cluster had opinions about that. Now they’re on the same page.

---

The takeaway

Eleven milliseconds is long enough to lose quorum, long enough to trigger failover, and long enough to create a data integrity incident. Your power infrastructure and your distributed systems are part of the same system—even if different teams own them.

Build your tolerance budgets accordingly.