Field Notes

Fundamentals before tools

Every tool in your lab will be replaced within a few years. The ideas underneath them will not. When the friendly interface fails, the people who understand those ideas fix the problem in minutes, and everyone else waits.

For several years I taught Linux administration to working adults. The students who struggled most were rarely the beginners. They were the people who had used a particular management interface for years and had never needed to know what it did on their behalf. They could click through any task the interface offered. The moment it showed an unhelpful error, they were stuck, because they did not know what the interface had been doing in the first place.

That pattern shows up in homelabs constantly. Someone builds an impressive stack from a tutorial, it runs beautifully for months, and then an update changes one thing. The dashboard turns red. The logs mention a mount point. And because the stack was assembled rather than understood, the only option is to search for someone with the same error and hope.

A storage pool that did not come back

A friend of the site runs a small NAS managed through a polished web panel. After a routine kernel update, the panel showed the pool as unavailable and offered no explanation. The panel could not say why, because the reason lived below it. At a terminal, two commands showed that one drive's controller had come up under a different device name and the pool simply needed to be imported again. Ten minutes, start to finish. The panel was not bad. It just was not designed for the one situation where you need help most.

Web interfaces are wonderful. Use them. But you should know what they do underneath, so that when they cannot help you, you still can.

The fundamentals I would learn first

If I were starting a homelab today, this is the list I would work through before chasing any particular product:

  • Addressing and routing. What an IP address, subnet mask, gateway and route are. Why a device with a 169.254 address did not get a DHCP answer. How a firewall decides what to forward.
  • DNS and DHCP. How a name becomes an address, who hands out addresses and how to query a specific server with dig. A surprising share of outages, at home and at work, end up here.
  • VLANs. Tagged and untagged ports, trunks and why a VLAN on its own is not a security control.
  • The shell. Moving around, reading files, managing services, reading logs, editing with vi. The shell is the lowest common denominator on every server, appliance and rescue disk you will ever meet.
  • Storage. Disks, partitions, filesystems, mounts, redundancy versus backups, and what a snapshot actually is.
  • Logs. Where they are, how to follow them while reproducing a problem and how to search them for the exact error.
  • Version control. Committing changes, seeing differences and rolling back. Git is not just for developers; it is the best undo button an administrator can have.
  • Hardware basics. Reading a hardware event log, running a memory test and recognizing when the problem is physical.

None of these belong to any vendor. They work on every distribution, every firewall, every hypervisor. They are what lets you pick up a new tool in an afternoon rather than a month.

Check the hardware before the code

One of the most expensive lessons in this hobby is spending weeks debugging software that is running on faulty hardware. Random crashes, corrupted files and errors that never repeat the same way are classic symptoms of failing memory, a dying power supply or a marginal cable. A memory test costs an evening. Skipping it can cost a month and, worse, silently corrupt data on its way to your storage.

Learn it by hand, then automate it

Automation is one of the most valuable skills in this field, and I write about it often. But automating something you do not understand produces a system nobody can repair. My rule for students and for myself:

  1. Build it by hand, following the documentation, and take notes on every step.
  2. Tear it down and build it again from your notes alone.
  3. Only then turn the notes into a script or configuration code.

The second manual build is where understanding actually forms. It is also where you discover which steps were unnecessary, which makes the automation simpler.

Plain text outlives everything

Keep your notes as plain text or Markdown files. I have notes from fifteen years ago that open perfectly, while notes I kept in two different proprietary applications are now in export formats that half work. Plain files can be searched with ordinary tools, read over SSH on a broken server and moved between any editors you like. Put them in git, synchronize them between your machines and you have a knowledge base that will last your whole career.

A word about assistants

AI assistants are now the first stop for many people with a technical question. They are genuinely useful for explaining unfamiliar output or drafting a first attempt at a script. They are also confidently wrong in exactly the areas where you cannot tell, and they tend to agree with whatever idea you bring them. Every skill you hand entirely to a machine is a skill you slowly lose. Use assistants the way you would use a colleague who is fast, well read and occasionally makes things up: as a starting point you verify, not an authority you obey.

The fundamentals are what let you do the verifying.

Where to start

If this list feels long, do not try to master it all at once. Pick a small project you genuinely want, such as a file server, a DNS filter or a monitoring dashboard, and learn whatever fundamentals that project touches. Then pick the next project. Our Start Here page suggests a path, and the Linux for the Homelab guide covers the shell skills that every other guide assumes.

The tools will change. You will still know what a gateway is.