U5 Operations

Automation and configuration management

Automation is not about saving time on day one. It is about capturing exactly how a system was built, so you can rebuild it after a failure, a migration or a year of forgetting. Servers become disposable; only the data and the code that builds them matter.

At a glance

  • Aim to delete any server and rebuild it with one command, then reattach its data.
  • Do it by hand once so you understand it. Automate it the second time.
  • Put configurations, scripts and notes in git. Never put secrets there, not even in a private repository.
  • Test changes on staging machines before production machines see them.
  • Automation multiplies mistakes as efficiently as it multiplies good changes. Keep a way back in.

The automation mindset

The core idea is simple: data is precious, servers are not. When a server's operating system, packages and configuration can be recreated from code, and its data lives on separate storage, a failed disk or a broken upgrade stops being a crisis. You rebuild, reattach the data and move on.

Automation also documents. A playbook or script is an exact, executable record of how a system was built. Notes written after the fact are always incomplete; code that built the system is not.

Rule of thumb

Automate anything you are likely to do again, anything tedious, and anything you would regret doing by hand during an emergency. Skip true one-offs, but be honest: the cluster you will never rebuild is usually rebuilt within the year.

A path that works

Do not start with the most sophisticated tools. Each stage builds the habits the next one needs.

  1. Write down every command you run while building something, in a notes file.
  2. Turn the notes into a shell script and run it on a fresh VM until it works from start to finish.
  3. Put the script under version control with git, so every change is recorded and reversible.
  4. Adopt a configuration management tool that describes the desired state instead of a sequence of commands.
  5. Add staging and automated testing once your fleet and your code have grown large enough to break each other.

Learning the manual way first is not wasted effort. You cannot debug automation for a system you do not understand.

Version control for everything

Git needs no server. Run git init in any directory and it starts recording changes locally. Start with your notes and service configurations:

cd /etc/reverse-proxy
sudo git init
sudo git add .
sudo git commit -m "Baseline configuration"

Afterward, git status and git diff show exactly what changed since the last commit, whether you changed it, a package update did or someone else in the house did. Rolling back a bad edit is one command.

Keep secrets out

  • Create an ignore file listing sensitive files before the first commit. Ignoring a file later does not remove it from history.
  • Store secrets in a separate encrypted file or a secrets manager, and reference them from your code.
  • Use different encryption keys for different projects, so one leak does not unlock everything.
  • If a secret is ever pushed to a public repository, treat it as stolen. Automated scanners find exposed credentials within minutes. Rotate it, then clean up the history.

Dotfiles

Keep shell and editor configuration in a repository in its own directory, and link the files into place with a small script. Do not turn your whole home directory into a repository; it is too easy to publish private keys that way.

Choosing the right tool for each job

StageWhat it doesHomelab notes
Image buildingProduces a template or golden imageKeep images minimal: user, SSH key, updates enabled. Let configuration management do the rest.
ProvisioningMakes infrastructure exist: VMs, networks, disksValuable for repeatable labs. Always read the plan of what will be created or destroyed before applying it.
Configuration managementBrings machines to a desired state and keeps them thereThe heart of homelab automation. Start here.
First-boot initializationSets hostname, user and keys, then hands offPerfect for single-board computer images and VM templates.
Continuous integrationTests changes automatically when you commitWorth it once you support several distributions or many machines.

Choose popular, well-documented tools with large communities. When you are stuck at midnight, the number of people who have already solved your problem matters more than elegant design.

Push or pull

Push

  • A control machine connects to each host over SSH and applies changes.
  • Nothing extra runs on the hosts.
  • Fails noisily for machines that are asleep or off.
  • Best for always-on servers and network devices.

Pull

  • Each machine periodically fetches a git repository and applies it to itself.
  • Works for laptops and lab machines that sleep at night.
  • No central control machine to protect.
  • Needs monitoring, because failures happen quietly on each machine.

Whichever you choose, write tasks to be idempotent: running them twice must produce the same result as running them once. A task that reports a change on every single run is a sign of a bug, such as a line being appended to a file again and again.

A first baseline

A good first automation project is a baseline applied to every machine. It is small, useful immediately and teaches the structure you will reuse:

  • Create your user account with your SSH public key and sudo rights.
  • Apply the SSH hardening settings from Linux for the Homelab.
  • Install and enable time synchronization. Clock drift breaks file sync, certificates and log correlation.
  • Enable automatic security updates with a notification.
  • Install your monitoring agent and a set of everyday command-line tools.

From there, add roles for specific machine types: servers, container hosts, workstations. Assign roles by hostname or group, and every new machine configures itself.

Stage before production

The more machines your automation manages, the more a single typo costs. A simple, effective arrangement:

  1. Keep two branches, staging and main.
  2. Keep one small reference VM per distribution you support, configured to follow the staging branch.
  3. Have each reference VM report success or failure to your phone after every run, including the error text.
  4. Merge to main only when every reference VM passes. Production machines follow main.

It is far better for four test machines to complain than for thirty production machines to lock you out. Snapshot reference VMs before testing major changes so you can revert and retry as often as you like.

Never fail silently

Scheduled automation fails quietly: a machine stops pulling, a job errors at 3 a.m., nothing tells you. Use a dead man's switch: at the end of each successful run, send a check-in to a monitoring service, and alert when an expected check-in does not arrive. This catches jobs that never start, which error alerts cannot. See Monitoring and Logging.

Automation risks

  • Mistakes scale. A wrong firewall rule pushed to every host can cut off every host. Roll out to a few machines first.
  • Keep a way back in. Out-of-band console access and backup codes for multi-factor authentication matter most on the day automation breaks access. See Out-of-Band Management.
  • Protect the automation itself. A machine that can change every server is the most valuable target in your lab. Keep it off the internet, use a dedicated account with only the permissions it needs, and do not reuse your personal key.
  • Do not over-architect. The simplest automation that rebuilds your systems is the best automation. Complexity is technical debt with a nicer name.

Caution

AI assistants are useful for drafting scripts and explaining unfamiliar syntax. Treat their output as a first draft from a confident stranger: read every line, test it on a disposable VM and never paste secrets or private configuration into a hosted service. We cover this in Local AI in the Homelab.