U5 Operations
Troubleshooting: a method for finding what broke
Most homelab problems are not mysterious. They are a loose cable, a full disk, a clock out of sync or a DNS record pointing somewhere unexpected. A calm, repeatable method finds them faster than intuition, and it works at 2 a.m. when intuition does not.
At a glance
- Work from the simplest, most common causes upward: power, cables, link lights, addresses, names, then software.
- Read the logs before you search the web, and search for the exact error line.
- Change one thing at a time and write down what you changed.
- Reproduce and experiment on a clone or snapshot, never on the only copy.
- Measure before and after. "It feels faster" is not a result.
The method
- Define the symptom precisely. Which device, which service, since when, for whom, and what exactly happens. "The internet is down" and "one laptop cannot resolve names on the guest network since this morning" lead to very different places.
- Ask what changed. Updates, new devices, configuration edits, power events, a moved cable. Your change log and version-controlled configs answer this quickly.
- Check the bottom layers first. Power, cables, link lights, negotiated speed, IP address, gateway, DNS. Most faults live here.
- Read the logs on both ends: the client and the server, the source and the firewall in between.
- Form one hypothesis and test it with one change. Undo the change if it did not help.
- Write down the fix, and what you would monitor to catch it earlier next time.
Rule of thumb
When a problem is intermittent, suspect the physical layer, timing or capacity: a marginal cable, a failing optic, a clock drifting, a disk or log volume filling, a battery aging. Intermittent software faults exist, but they are rarer than people assume.
Start at the physical layer
- Look at the link lights and the negotiated speed on the switch. A gigabit port linked at 100 Mb/s usually means a damaged cable or a bad termination.
- Swap the cable before you swap anything else. Bad bulk cable can produce months of mysterious failures.
- Reseat optics and direct-attach cables, and keep spares. A failed transceiver can look exactly like a cut fiber.
- Check power. An underpowered single-board computer or a failing power brick causes random crashes and corrupted storage.
- Think about the environment. Heat, a chair rolling over a cable, a new appliance near an access point.
Addresses, routes and names
Once the link is up, work through the network in order:
| Question | How to check | Common culprit |
|---|---|---|
| Did the device get the right address? | ip addr on Linux; the DHCP lease table on the firewall | Wrong VLAN on the port; a 169.254 address means no DHCP answer |
| Can it reach its gateway? | ping the gateway address | VLAN mismatch on a trunk; firewall rule on that interface |
| Can it reach an outside address? | ping a well-known public IP | Missing outbound rule or NAT problem |
| Can it resolve names? | dig a name against your resolver, then against another | Resolver down, forgotten VLAN not allowed to query, stale record |
| Is the service listening? | ss -tlnp on the server | Service bound to localhost only, or not running |
| Is something in between blocking it? | Firewall logs with logging enabled on block rules | A rule order problem, or the host firewall |
There is a reason for the old joke about DNS. When some things work and others do not, query the same name against different resolvers with dig and compare the answers.
Read the logs
journalctl -u name.service -ffollows one service's log while you reproduce the problem.journalctl -b -p errshows errors since the last boot.dmesgshows kernel messages, including disk, memory and network card errors.- The authentication log shows why an SSH login failed, which the client deliberately does not tell you.
- A terminal log viewer that colors and merges several logs makes patterns easier to see.
When you search for help, use the exact error text and include versions. When you ask a person or an assistant for help, share only the relevant lines and remove secrets.
Storage problems
- Disk full, but free space shows available? Check inodes with
df -i. Millions of tiny files can exhaust them while gigabytes remain. - Find what is using space with an interactive disk usage tool. Install it before you need it, because you cannot install software on a full disk.
- Service fails after a reboot? Check that network shares and datasets actually mounted. An application writing to an empty mount point can fill the root disk or appear to lose all its data. Shared Storage for Containers shows how to prevent it.
- Drive errors? Check drive self-test results and the pool status. Intermittent errors that move between drives point toward a cable, backplane or controller, not the drives.
Separate hardware from software
Keep a bootable USB drive with a live Linux environment, a disk imaging tool and a memory tester. If a problem follows you into the live environment, it is almost certainly hardware. If it disappears, it is the installed software or its configuration.
From the Bench
A workstation on the Bench produced random application crashes for two weeks. We reinstalled software, changed drivers and blamed an update. An overnight memory test found a failing module in the first hour. We now run a memory test before blaming any software for random crashes.
Experiment on a copy
Production is the wrong place to experiment. Snapshot or clone the VM, reproduce the problem on the clone and break it as many times as you need to until you find the fix. Then apply the fix to production deliberately. Restoring last night's backup into an isolated clone has a bonus: it tests your backup at the same time.
Give snapshots descriptive names, and be careful with identical shell prompts on the original and the clone. More than one administrator has applied the experimental fix to the wrong machine.
Measure, do not guess
Performance problems need numbers. Before changing anything, record a baseline. Then change one setting and measure again.
- Network throughput. Use a raw network throughput test tool between the two machines before blaming storage. A link that tests at 940 Mb/s on a supposedly 10 GbE connection is plugged into a gigabit port somewhere.
- Storage. Measure sequential and random performance separately. Fast sequential numbers say little about how VMs will feel.
- Real workloads. Copy the same large test file each time, and watch real-time metrics while you do. Asymmetries between reading and writing are often the clue.
- Record everything. A short note of each change and its result keeps you from repeating tests and from fooling yourself.
Know when to stop
If a fix is not in sight after a reasonable effort and you have good backups, restoring a known-good state is a legitimate answer. Keep the logs and a copy of the broken system if you want to find the root cause later, but get the household working again first.
And when you do find the cause, write it down. The next time it happens, your notes will find it faster than your memory. See Documentation and Technical Debt.