A rebuild window is a security window

The rebuild that never ends

You’re running RAID-Z on your ZFS array. A drive dies at 3 AM. You replace it, and now you’re watching zpool status show the rebuild progress at 41%.

That’s when someone asks you to pull a dataset for testing.

The exception that became the rule

We had four degraded arrays in production. Each one “temporary.” Each one forgotten. Until the fourth one lost a second drive before the first could finish rebuilding.

# Check your degraded pools right now
sudo zpool status | grep -A 5 'DEGRADED'
# If that returns anything, you have work to do

The security implications nobody talks about

A degraded array isn’t just a reliability problem—it’s a security window:

1. Reduced fault tolerance: One more failure and you’re gone
2. Performance degradation: Queries slow down, timeouts increase
3. Operational pressure: “Can we just push the upgrade to next week?”
4. Human error: Fatigued operators make mistakes

What we changed

After incident #4, we made rebuilds a priority, not an afterthought:

• No new deployments during a rebuild
• Automatic alerts when any pool drops below optimal
• Scheduled maintenance windows dedicated solely to recovery
• Capacity planning that accounts for rebuild headroom

---

The numbers that matter

Metric	Before	After
Avg rebuild time	18 hours	12 hours
Pools degraded >24h	4	0
Secondary failures during rebuild	1	0

The rebuild window is a security window. Don’t let it slip away.