U5 Operations
Documentation and technical debt
Six months from now you will not remember why that firewall rule exists, which drive bay holds the oldest disk or how you set up the reverse proxy. Write it down now, in a format you can still read in ten years.
At a glance
- The best documentation tool is the one you will actually use. Writing something down matters more than where.
- Keep notes in plain text or Markdown, under version control, with secrets kept elsewhere.
- Document the current state, the intended future state and the steps to rebuild everything.
- Label every cable, port and drive bay. Physical labels are documentation you cannot misplace.
- Technical debt is anything you are afraid to update. Rebuild it until updating is routine.
Why document a home lab
Documentation is how you talk to your future self, and to anyone who has to keep the house running if you are away. It turns a two-hour outage into a ten-minute fix, it is the raw material for automation, and it is what you will want in hand when you ask for help. Drawing a diagram of your network to explain a problem to someone else often reveals the problem before you finish.
Documentation also has career value. A public, tidy write-up of how you built your lab says more about you to a hiring manager than a line on a resume.
What to write down
| Document | Contents |
|---|---|
| Inventory | Every device: model, serial number, location, purchase date, warranty, firmware version |
| Network map | Networks and VLANs, address ranges, gateway, DHCP ranges, reservations, switch port assignments |
| Diagrams | One physical diagram (what plugs into what) and one logical diagram (networks, services, flows) |
| Service catalog | Each service: where it runs, how to reach it, where its data lives, how it is backed up, who uses it |
| Runbooks | Step-by-step procedures: restore from backup, replace a failed drive, rebuild the firewall, recover from a power cut |
| Change log | Dated one-line entries: what you changed and why |
| Recovery sheet | Where the password manager recovery kit, backup encryption keys and hardware security key spares are kept |
Write two versions of the architecture: the lab as it is today and the lab you are working toward. The gap between them is your project list.
Rule of thumb
If you had to search for how to do something, write down how you did it, right then. The command you look up twice is a note. The note you use three times is a script.
Plain text, under version control
Formats outlive tools. Notes kept as plain Markdown files in folders will open in any editor in twenty years, can be searched with ordinary tools, synchronize easily and work over SSH on a broken server. Proprietary note databases and hosted platforms come and go, and some change their terms or start mining what you write.
- Put the notes folder in a git repository. You get history, the ability to see what changed and a natural way to learn git with nothing at stake.
- Synchronize the folder between your machines with a file sync tool, so the note you typed a minute ago survives a dead laptop.
- Keep secrets out. Passwords, API keys and private keys belong in your password manager. Reference them by name in your notes.
- Choose an editor you like. Any Markdown editor works on plain files, and you can switch later without migrating anything.
Wikis are excellent for households or groups that document together. Prefer simple ones with low maintenance, and remember that a self-hosted wiki is one more service to patch. Flat-file wikis with no database are the easiest to back up and restore.
Diagrams that help
A good diagram is simple. Use a proper diagramming tool rather than a paint program, keep crossing lines to a minimum and label every box with its name and address. Annotate what does not connect as well as what does, because that is often the part people get wrong. Export diagrams to an image and keep the source file next to your notes.
Labels: documentation you can touch
- Label both ends of every cable with the same identifier.
- Label switch ports, or keep a printed port map inside the rack door.
- Label every drive bay with the drive serial number, and record the same mapping in your inventory. When a pool reports a failed drive by serial, you will know which bay to pull.
- Label power cords at the plug end, so you never unplug the wrong device.
- Mark the date on anything with a battery: UPS units, firmware batteries, sensors.
Runbooks for bad days
A runbook is a procedure written for someone tired and stressed, which on the day you need it will be you. Write it as numbered steps, include the exact commands and where to find credentials, and test it. Keep printed copies of the most critical runbooks, such as restoring the firewall and recovering backups, because you may need them when the network that holds your notes is down.
Understanding technical debt
Technical debt is the cost you pay later for shortcuts taken now. In a homelab it looks like this:
- A service built by hand three years ago that you have never updated because the install was painful.
- A temporary firewall rule that became permanent.
- A container whose data location you are not sure about.
- Drives with five or more years of power-on time holding data with no second copy.
- An operating system release that has reached the end of its support window.
- A cluster nobody, including you, fully understands anymore.
Debt is not just untidiness. It is a security problem. Systems that are painful to update do not get updated, and outdated internet-facing systems are how networks are breached. Commercial products are not immune: closed appliances have shipped with components more than a decade out of date behind a refreshed interface.
Caution
Complexity is debt too. Deploying a platform you do not understand, because a tutorial made it look easy, creates a system you cannot repair when it breaks. Prefer the simplest design that meets your needs, and grow into complexity only as your understanding grows.
Paying it down
- List it. Go through your service catalog and mark anything that is out of support, undocumented, hand-built or frightening to update.
- Prioritize by exposure and value. Anything reachable from the internet first, then anything holding irreplaceable data.
- Rebuild in parallel. Build the replacement alongside the old system, copy the data across, test, then switch over. Keep the old system until you are sure.
- Capture the rebuild as notes, then as a script or configuration code.
- Repeat until it is boring. When rebuilding a service is routine, updating it stops being scary.
From the Bench
Our password vault ran on an operating system release that was nearing the end of its support. We built a new VM on the current release beside it, copied the data directory and two configuration files, tested logins from every device, then moved the DNS name across. Total downtime was about five minutes, and the runbook we wrote along the way is now a script.
The homelab is the ideal place to build this habit. At home you can delete a service and rebuild it as many times as you like, until the process is in your muscle memory. That confidence is exactly what lets you remove old systems at work instead of living with them.