Field Notes
The restore nobody tested
The backup reports were green every morning for four years. Then the office flooded, and we discovered that the one thing needed to read those backups had been underwater with everything else.
Before About Homelabs, I spent many years looking after small offices: architects, accountants, a veterinary practice, a few nonprofits. Most of the lessons in our backup guide were learned there, and one of them cost a client more than any other.
Four years of green checkmarks
The client was a small design practice with a single server in a closet. We had set up what looked like a model backup arrangement: nightly snapshots on the server, a nightly copy to a second device in the office and an encrypted weekly copy to cloud object storage. Every morning a report arrived saying everything had succeeded. Every morning, for four years, it did.
Then a pipe burst on the floor above over a long weekend. The closet held the server, the second backup device and, as it turned out, the only copy of the encryption passphrase for the cloud backups. Someone had saved it years earlier in a text file on the server's desktop, "temporarily", so the backup software could be reconfigured after an update.
The cloud copy was complete, current and perfectly encrypted. Without the passphrase it was noise. We recovered some of the practice's work from staff laptops and email attachments. The rest was gone.
No component in that system failed. The backups ran, the encryption worked and the cloud provider kept the data safe. What failed was that nobody had ever tried to restore from the off-site copy alone, as if the office no longer existed. A single drill would have asked the question that mattered: where is the key?
Our own first drill
When we built the Bench, I was determined not to repeat that mistake. We wrote a restore runbook on the first day. It took two more years before we ran a full drill against it, and I am a little embarrassed by what we found:
- A dataset was missing from the off-site copy. We had added a new dataset for photos eight months earlier. The local backup picked it up automatically; the off-site job had an explicit list of datasets and never had.
- The key was in the wrong place. The off-site encryption passphrase was in our password manager, which was correct. But the password manager's own server ran as a VM on the hypervisor we were pretending had been destroyed. The printed recovery sheet was in a drawer, and it was out of date.
- The restore took far longer than we assumed. Downloading the off-site copy over our home connection took most of a day. Our plan had silently assumed an hour.
- The runbook had drifted. Two steps referred to software we no longer used, and one assumed a network layout we had since changed.
Nothing was lost, because it was a drill. That is the whole point of a drill.
What we changed
- Off-site jobs now back up the whole pool by default, with explicit exclusions instead of explicit inclusions.
- Recovery material lives in three places: the password manager, a sealed printed sheet at home and a second sealed copy at a family member's house. We review it every January.
- We keep a small encrypted export of the password vault outside the lab, so the vault is not trapped inside the system it is meant to help recover.
- Our runbook records how long each step actually took in the last drill, so our expectations are based on evidence.
- The drill is on the calendar, once a year, and a smaller restore test happens every month.
How to run your own drill
Set aside a full weekend day; downloading an off-site copy takes longer than you expect. Then pretend the house is gone.
- Use a different machine. A laptop that is not part of your lab, or a borrowed one.
- Use only what would survive. Your off-site copy, your printed or remote recovery material and your written runbook. No peeking at the NAS.
- Restore something real. A folder of family photos, a password vault export or a complete virtual machine.
- Open what you restored. A restored file you have not opened is not yet a restored file.
- Time every step, and write down every point where you hesitated, searched or guessed.
- Fix what you found and update the runbook before you forget.
Your first drill will find something. Mine did. Better a drawer full of out-of-date notes on a quiet Saturday than a closet full of water on a Monday.
Our Backup Strategy guide covers the full strategy, and the backup verification checklist turns the routine into a printable list.
Nobody cares about backups. Everybody cares about restores.