U5 Operations
Monitoring and logging: know before the household does
The most dangerous failure in a homelab is the quiet one: the backup that stopped in March, the disk that filled overnight, the sync job that died a week ago. Good monitoring is less about pretty dashboards and more about never being surprised.
At a glance
- Start with two layers: an uptime dashboard that checks services and a real-time metrics agent on each host. Add heavier tools only when you need them.
- Monitor absence as well as failure. Scheduled jobs should check in when they succeed, and you should be alerted when they do not.
- Send alerts to a dedicated channel that you actually see, not to an inbox you skim.
- Centralize logs with sensible retention, consistent time zones and synchronized clocks.
- Watch the watcher: something outside your lab should notice when your monitoring itself goes down.
Decide what matters
Monitoring everything produces noise. Monitoring nothing produces surprises. Start from the question: what would someone in the house notice first if it broke? Usually that is internet access, wireless, DNS, the media library, the photo backup and the password vault. Then add what would hurt most if it failed silently: backups, sync jobs, drive health and free space.
| Signal | Example checks | Alert when |
|---|---|---|
| Availability | Web services respond, DNS resolves, the firewall answers ping | A check fails several times in a row |
| Capacity | Disk and pool usage, memory, swap | Pools pass 80 percent; swap use climbs steadily |
| Health | Drive self-tests, scrub results, temperatures, UPS state | Any error count increases; the UPS switches to battery |
| Jobs | Backups, replication, sync, certificate renewal, configuration runs | An expected check-in does not arrive |
| Security | SSH logins, admin logins, firewall changes, new devices on the network | Anything happens that you did not do |
Myth
"High CPU means trouble." A server at 60 to 75 percent CPU is simply working. Worry when CPU sits at 100 percent and load keeps rising, when memory pressure forces heavy swapping, or when a normally busy system suddenly goes quiet.
Layer one: uptime checks
A self-hosted uptime dashboard checks whether services answer: an HTTP request returns the expected page, a TCP port accepts connections, a keyword appears. It is light enough to run on a single-board computer, simple to configure and gives the household a status page. It does not tell you why something failed, only that it did. Up and working are not the same thing, so check for meaningful responses, not just open ports.
Layer two: real-time metrics
Metrics agents collect CPU, memory, disk, network and application statistics from each host. Modern agents discover what is running and configure themselves, and some collect every metric every second. That granularity matters: tools that sample once a minute and average the results can hide stalls of a second or two that users definitely feel.
- Use real-time metrics to watch a problem as it happens, for example while copying a large test file between machines and changing one setting at a time.
- Use longer-term history to spot trends: a disk filling a little more every week, memory use creeping up after each update.
- Know your baseline. Anomalies are only meaningful against what normal looks like for your lab.
Full dashboard stacks with separate collectors, databases and visualization layers are powerful and common in industry. They are also several systems to maintain. Start with the simpler layers and graduate when you have a specific question they cannot answer.
Catch silent failures
A job that fails loudly is easy. A job that never runs at all sends no error to alert on. The answer is a dead man's switch:
- Give every scheduled job a unique check-in address on a monitoring service, self-hosted or hosted.
- Make the job call that address as its very last step, only if everything before it succeeded.
- Set an expected interval, such as one day for a nightly backup plus a grace period.
- Alert when a check-in is missing. You learn that the job stopped, whatever the reason.
Apply the same idea to logs and metrics: a log source that goes silent, or a metrics agent that stops reporting, deserves an alert. Silence is not health. Monitoring catches problems as they happen; a short maintenance routine catches the slow decay that alerts miss.
Central logging
When something goes wrong, the answer is usually in a log. When the logs are scattered across twenty machines, you hop from one to the next and lose the thread. A central log server collects them in one place, parses them into searchable fields and lets you correlate events: the VPN login, the firewall change and the service restart that happened in the same minute.
Practical setup
- Send firewall, switch, access point, hypervisor, NAS and server logs to one log server.
- Synchronize every clock and set time zones deliberately on the log server and its sources. Mismatched time zones make fresh logs appear to be missing.
- Use a separate index or retention rule per source. Firewall logs are often most of the volume; storage and hypervisor logs are small but valuable.
- Keep log data on its own volume, separate from the operating system disk, and alert before it fills. A full log disk can stop logging silently.
- Carry log traffic on the management network, and use encrypted transport where your devices support it.
How long to keep logs
Retention is a trade-off between disk space and the questions you may want to answer later. A useful default for a home: firewall and authentication logs for 90 days or more, everything else for 14 to 30 days. Longer firewall retention lets you answer a question that comes up after every major vulnerability announcement: did anything in my network talk to these addresses before I patched?
From the Bench
Our Bench produces roughly 4,000 log messages a minute, and about 90 percent of them come from the firewall. Thirty days of firewall logs take around 45 GB. Thirty days of logs from every switch and access point combined take well under 1 GB.
Alerts that reach you
An alert you do not see is worse than no alert, because it creates confidence you have not earned.
- Use a dedicated channel, such as a push notification app or a chat channel used only for alerts. If it buzzes, it matters.
- Do not run your own mail server just for alerts. Use authenticated sending through a hosted mail provider if you want email.
- Tune ruthlessly. Every alert that fires without needing action teaches you to ignore the next one. Raise thresholds, add delays or delete checks that cry wolf.
- Alert on security signals that are rare at home. In a lab with one administrator, any SSH login you did not make is worth an immediate alert.
- Include context. The host, the check, the value and a hint about what to look at first.
Watch the watcher
If your monitoring server shares a power circuit, a switch or a hypervisor with everything else, it fails at the same moment and tells you nothing. Add one check from outside the lab: an external service, or a small instance on a rented server, that confirms your monitoring is alive and your internet connection is up. Have that external check alert through a different path from your internal alerts.
Power and environment
- UPS state. A UPS monitoring service can report battery status, alert when mains power fails and trigger orderly shutdowns. See Racks, Power and Cooling.
- Temperature. A cheap sensor in the rack or closet catches a failed fan or a closed vent before hardware does.
- Water. A leak sensor under or near the rack costs little and has saved more than one lab from a burst pipe or a failed water heater.
Detection tools are learning tools
Network intrusion detection, security monitoring suites and host-based detection agents teach you what a security operations team actually does. Expect a flood of alerts at first, most of them false positives. Run them in alert-only mode, read and tune the rules, and treat the work as education. High-fidelity tripwires, such as a decoy file or credential that nothing legitimate should ever touch, are far more useful at home than a noisy detector. More in Securing Your Homelab.