Fixed · Incident
When the database lost its primary
Every node up, none accepting writes. Restored a healthy primary and wrote the runbook.
- 01
Symptom
Writes started failing although every database node was running.
- 02
Root cause
The replica set had no electable primary, and the cache was writing to a node that had become read-only.
- 03
Fix
Inspected member state, restored a healthy primary, and made the cache follow failover instead of a fixed node.
- 04
Impact
Writes restored, with a recovery runbook for next time.
Debug log
- $check the status of every member
- $fix what blocked the election
- $make the cache follow failover
- $write the runbook
- ✓resolved
A thousand welcomes