U4 Compute

Linux for the homelab: the operating system under everything

Nearly everything in a homelab runs on Linux, including most appliances that hide it behind a web interface. You do not need to master it all at once. You need a sensible distribution, good SSH habits, a patching routine and a few tools you can rely on when something breaks.

At a glance

  • For servers, choose a stable or long-term support release. Use rolling releases where you want new software and can tolerate more maintenance.
  • Use appliance distributions for firewalls, NAS boxes and hypervisors, and do not customize them into something else.
  • Keep a second distribution tested and ready. Vendors change direction without asking you.
  • Use SSH keys with passphrases, disable password and root logins, and keep SSH off the internet.
  • Patch automatically, reboot on a schedule and treat a long uptime as a warning sign, not a trophy.

Choosing a distribution

A server distribution and a desktop distribution share the same kernel; the difference is what is installed. Start every server from a minimal install and add only what it needs. Fewer packages means fewer updates, fewer vulnerabilities and less to break.

Release models

ModelHow it worksBest for
Conservative stableA new release every two years or so; security fixes only in between; software versions stay fixed for the life of the releaseServers you want to forget about
Long-term support (LTS)A stable release with a published support window, often five years for server use, plus newer kernels for recent hardwareServers on newer hardware, mixed fleets
Short-cycle releasesA new release every six to nine months, each supported brieflyDesktops that want newer software; not ideal for servers
RollingContinuous updates with no version upgradesWorkstations and enthusiasts; servers only with minimal packages and snapshots before updates

Conservative distributions sometimes lag on support for brand-new hardware, such as the newest network cards. LTS releases that ship newer hardware-enablement kernels solve this. Check hardware support before committing a new machine, by booting a live image and confirming every device is detected.

Appliance or general-purpose?

Firewall, NAS and hypervisor distributions are Linux or BSD underneath, but they are appliances: curated packages, a web interface, one configuration file to back up and a tested upgrade path. That is a feature. The moment you add third-party repositories or packages to an appliance, you have forked it, and you become its only support team. Small additions such as a monitoring agent are usually fine. Anything bigger belongs in a VM or container.

Keep a second distribution

Distributions change direction: support windows shrink, source access changes, release models shift. If all your automation assumes one distribution, any such change becomes an emergency. Keep your automation and container hosts working on a second distribution, ideally from the same package family, and test it now and then. Our field note Fundamentals Before Tools explains why skills that transfer between systems matter more than mastery of one.

SSH done properly

SSH is the management plane of your lab. It deserves careful setup.

Keys

Generate a separate key for each trust domain, such as one for the lab and another for work, with a descriptive filename and a passphrase:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_lab -C "lab key"
ssh-copy-id -i ~/.ssh/id_ed25519_lab.pub hv01.lab.example.net

A passphrase protects the key if your laptop is stolen; an SSH agent caches it so you type the passphrase once per session. Never reuse one key everywhere. If a key ever leaks, remove its public half from every server, passphrase or not.

Client configuration

A client config file turns long commands into short aliases and makes sure the right key is used for each host:

Host hv01
    HostName hv01.lab.example.net
    User yourname
    IdentityFile ~/.ssh/id_ed25519_lab
    IdentitiesOnly yes

The IdentitiesOnly setting matters once you have several keys. Without it, the client may offer every key in turn and the server will disconnect you for too many failed attempts before reaching the right one.

Server hardening

Once key login works, tighten the server. These settings in the SSH server configuration cover the essentials:

PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
AllowUsers yourname

Caution

Before restarting the SSH service, test the configuration with sudo sshd -t and keep your current session open. Open a second session to confirm you can still log in. Only then close the first. Everyone locks themselves out once; this routine makes sure it happens on a lab box and only once.

Add a tool that bans addresses after repeated failed logins, and keep SSH behind your VPN. Changing the port reduces log noise from automated scanners but is not a security measure on its own. See Remote Access and VPNs for jump hosts and remote access.

When a key login fails

The SSH client tells you very little on purpose. Run the client with -v to see which keys it offers, and watch the server's authentication log while you retry. The usual causes are private key permissions that are too open (the file should be readable only by you), the wrong key being offered, or the wrong username.

Patching without fear

Unpatched systems are how homelabs get compromised. The fear that updates will break things is mostly inherited from other platforms; on a stable Linux release, security updates are small and conservative.

  • Enable automatic security updates on every server, with an email or notification summarizing what was installed.
  • Prioritize internet-facing systems, then everything else. Internal does not mean exempt.
  • Know what needs a reboot. Updated services only need restarting; a new kernel needs a reboot to take effect. Updated shared libraries affect everything that uses them, so rebooting is often simpler than hunting for affected processes.
  • Schedule reboots at a quiet time. For lab machines that sleep at night, patch shortly before the scheduled shutdown so they wake up updated.
  • Snapshot before major changes so a bad update is a rollback, not a rebuild.
  • Keep a fallback kernel. On stable distributions the previous kernel stays installed and selectable at boot. On rolling distributions, install a long-term kernel alongside the main one.

Rule of thumb

A broken update is an afternoon of work. A compromised server is a rebuild, a password reset for everything it touched and a question about what else was reached. Patch.

systemd features worth knowing

Most people use systemd only to start, stop and restart services. A few more features solve real problems.

Override instead of editing

Never edit unit files installed by packages; an update will overwrite your change. Use sudo systemctl edit name.service instead. It creates a small override file containing only your changes, which survives updates. Unit files you write yourself belong in /etc/systemd/system, which takes priority over packaged units.

Find slow boots

systemd-analyze blame lists services by how long they took to start, slowest first.

Read logs

journalctl -u name.service -f follows the log of one service. Add -b to see only messages since the last boot, or -p err to see only errors.

Timers instead of cron

A timer is a pair of units: a service that does the work and a timer that schedules it. Timers log to the journal and can run a job that was missed while the machine was off. A nightly job looks like this:

[Unit]
Description=Nightly configuration export

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

Save it as /etc/systemd/system/config-export.timer next to a matching config-export.service, then enable it with sudo systemctl enable --now config-export.timer. The Persistent setting means a run missed during a power outage happens at the next boot.

Small skills that pay off

  • Learn basic vi. It is present on virtually every system you will ever need to repair, including appliances, when no other editor is.
  • Use a terminal multiplexer on a jump host or always-on machine so long-running work survives a dropped connection.
  • Read the logs before searching the web. Search for the exact error line, not a description of the symptom.
  • Use universal package formats sparingly. They are handy for a few fast-moving desktop applications. On servers, prefer the distribution's own packages or containers.
  • Keep notes. Every command you had to look up is worth writing down. See Documentation and Technical Debt.