U5 Operations
A maintenance routine for your homelab
Labs rarely fail all at once. They decay quietly: a backup stops, a disk fills, an update waits for months. Ten minutes a week and an hour a month catch almost all of it. This is the routine we follow on the Bench.
At a glance
- Automate the daily checks and let monitoring tell you when something needs attention.
- Spend about ten minutes a week confirming services, backups and free space.
- Spend about an hour a month on updates, accounts, drive health, cleanup and a small restore test.
- Follow the same short ritual before every major update.
- Keep a service catalog, and update it whenever you add or remove something.
Why a routine beats heroics
The most expensive homelab problems are the ones that went unnoticed for weeks: a sync job that silently stopped, a pool that crept past 90 percent, a service left on a version with a published vulnerability. None of these are hard to fix early. A routine turns them from emergencies into chores.
The routine also stays small because most of the work is automated. Monitoring and dead man's switches watch continuously, as described in Monitoring and Logging. Your job is to confirm that the automation is still working and to handle the things it cannot.
Every day: automated
- Uptime checks on every service the household uses.
- Check-ins from every scheduled job, alerting when one is missing.
- Alerts on low disk space, drive errors, high temperatures and UPS events.
- A short daily health report, if you have built one. Ours arrives at five in the morning and lists memory use, running VMs against the expected count, backup age, failed tasks and free space.
Keep alerts meaningful. If an alert fires and you do nothing, change or delete it. Alerts that are routinely ignored train you to ignore the important ones.
Every week: about ten minutes
- Glance at the dashboard. Confirm every critical service is up and nothing has been flapping.
- Check the backups. Every job ran, sizes look normal and the newest backup is recent.
- Check free space on pools, VM storage and log volumes.
- Skim for errors. Look through the past week's warnings and errors in your central logs, or at least on the firewall and storage.
- Note anything odd in your change log for next month.
Every month: about an hour
- Apply updates that are not automatic: firmware, appliance releases and application containers, one at a time, using the ritual below. Security releases for the firewall and other internet-facing devices do not wait for this session; apply them within days.
- Review administrative accounts and SSH keys; remove anything unused.
- Check drive health, including SSD wear indicators. Hypervisors and log-heavy services write a lot; wear is easier to plan for when you watch it.
- Clean up. Remove unused containers, old images, stopped experiments and expired snapshots.
- Update the service catalog with anything that changed.
- Run a small restore test. Restore a few files or a small container's data and open them.
- Review external exposure. Confirm remote access settings and published services are what you intended.
- Check the UPS self-test result and battery status.
Every quarter and every year
Each quarter, run the full audit on our quarterly audit checklist and restore one complete VM or container stack to an isolated network. Once a year, run a full restore drill from the off-site copy, review hardware age and plan replacements, confirm the domain is set to renew automatically, review any software licenses and subscriptions, and refresh your documentation. See The Restore Nobody Tested for why the yearly drill matters.
Before every major update
Most self-inflicted outages come from applying a major update in a hurry. This short ritual prevents nearly all of them:
- Read the release notes, looking for breaking changes, removed features and required migration steps.
- Confirm a current backup exists, and that you know how to restore it.
- Take a temporary snapshot of the VM, container or dataset.
- Update one service at a time, never everything at once.
- Test what the household uses, not just whether the service starts.
- Record the change in your change log, then remove the snapshot once you are confident.
Caution
Automatic updates are excellent for operating system security patches. For major application versions, automatic updates without a rollback plan are how a working service becomes a broken one overnight. Let tools notify you of new versions, then update deliberately.
Keep a service catalog
A service catalog is the single most useful document in a homelab. It turns "what is this container?" into a thirty-second lookup and makes every routine check faster. Keep it in your notes repository, and never store passwords in it.
| Service | Runs on | Address | Data lives in | Backup | Updates |
|---|---|---|---|---|---|
| Password vault | Container host VM | vault.lab.example.net | /srv/vault | Nightly, off-site weekly | Monthly, manual |
| Photo library | Container host VM | photos.lab.example.net | NAS dataset for photos | Nightly, off-site weekly | Monthly, manual |
| Home automation hub | Dedicated VM | hub01.lab.example.net | VM disk | Daily | Monthly, manual |
| DNS filter | Firewall | 10.42.10.1 | Firewall configuration | Export after changes | With firewall updates |
Give every server a DHCP reservation or static address from day one and record it here. Services that refer to each other by address break in confusing ways when addresses change. More on documentation in Documentation and Technical Debt.
Make it stick
- Put the weekly and monthly sessions on your calendar, like any other appointment.
- Use the printable weekly and monthly maintenance checklist so you do not have to remember the steps.
- Keep a short log of each session. It becomes the history you consult when something breaks.
- Automate any step you find yourself doing the same way every time. See Automation and Configuration.
- Before a long trip, run the weekly check early and make sure the household knows what to do if something goes down.