U1 Foundations
Out-of-band management: remote hands for your lab
Out-of-band management gives you a keyboard, a screen and a power button for a machine whose operating system or network has failed. It is the difference between fixing a problem from your desk and fixing it from a cold basement floor with a borrowed monitor.
At a glance
- Out-of-band access works independently of the host operating system and its network connections. You can watch it boot, change firmware settings and reinstall it remotely.
- Most servers include a management controller. Desktops and mini PCs can get the same abilities from a small KVM-over-IP device.
- Serial consoles on firewalls and switches are simple, cheap and still the most dependable way in when the network is broken.
- Management interfaces must live on an isolated network and must never be reachable from the internet.
What out-of-band means
In-band management uses the same operating system and network as the services you run: you connect over SSH or a web interface on the machine itself. When that operating system crashes, a network card drops off the bus or a firewall rule locks you out, in-band access disappears with it.
Out-of-band management uses a separate path. It might be a small dedicated computer on the server's motherboard, an external KVM-over-IP device plugged into the video and USB ports, or a serial cable. With it you can:
- See the console during boot, including firmware screens and kernel errors.
- Power a machine on, off or through a hard reset.
- Change firmware settings and update firmware without attaching a monitor.
- Mount an installer image remotely and reinstall an operating system.
- Read hardware event logs that record memory errors, fan failures and power problems.
From the Bench
One of our hypervisor hosts started dropping its 10 GbE card at irregular intervals, always late in the evening. Over the network the host simply vanished. The management console showed the kernel error on screen and let us log in locally, collect logs and reboot cleanly, all without walking downstairs. That one evening justified the effort of setting it up.
Built-in management controllers
Most servers and many workstation-class motherboards include a baseboard management controller, usually called a BMC. It has its own network port or shares one of the host ports, its own IP address and its own web interface. The industry standards it speaks are IPMI, an older protocol for power control and sensor readings, and Redfish, a newer API built on HTTPS and JSON.
What to check on used servers
- Feature licensing. Some server vendors sell the remote console and virtual media as paid upgrades. Find out what the installed license includes and whether it is permanent.
- Console technology. Older controllers use browser plugins that modern browsers no longer run. Newer firmware offers an HTML5 console that works in any current browser. Two servers of the same model can ship with different controller generations, so check the exact firmware version.
- Firmware availability. Controller firmware has had serious security flaws over the years. Make sure you can download updates without a support contract.
- Dedicated port. A dedicated management port makes isolation much easier than a port shared with the host.
Rule of thumb
If an old controller only offers a plugin-based console, do not install the outdated runtime on your everyday computer. Run it inside a disposable virtual machine or container that you start only when you need the console and discard afterward.
KVM-over-IP for everything else
Mini PCs, desktops and consumer motherboards usually have no management controller. A KVM-over-IP device fills the gap. It captures the video output, emulates a USB keyboard and mouse, and serves the console to your browser. Many can also present an installer image as a virtual USB drive and connect to the motherboard's front-panel header for power control.
Open-source KVM-over-IP projects run on inexpensive single-board computers and cost far less than enterprise network KVMs. Pair one with a used multi-port KVM switch and a single device can reach eight machines. Before buying the switch, confirm that it supports keyboard shortcuts for changing ports, because that is how the KVM-over-IP device will switch between machines. Older servers with VGA output need an inexpensive HDMI adapter.
Serial consoles
Firewall appliances, managed switches and many servers expose a serial console. It is text only, it needs no network, and it works when almost nothing else does. A USB serial adapter and a terminal program are all you need. Keep one in the rack with a note of the correct speed setting for each device; 9600 or 115200 bits per second covers most equipment.
A small always-on machine with a few USB serial adapters makes an excellent console server. Combined with a VPN, it gives you a way into your firewall even when a bad rule has locked you out of its web interface.
Remote power control
Power control is the most used out-of-band feature. Management controllers can power a server on, off or through a reset, which also makes them the most reliable way to schedule servers to start in the morning. A small always-on device runs a scheduled job that sends the power-on command to each server's controller, avoiding the time zone and daylight saving problems of firmware power-on timers.
Switched power distribution units let you cut and restore power to individual outlets over the network. They are useful for appliances that hang hard, but a hard power cycle is a blunt instrument. Use it as the last resort, never as routine.
Keeping management interfaces safe
A management controller has complete control over the machine it manages. Anyone who reaches it can reinstall the operating system, read the console and change firmware. Treat management interfaces as the most sensitive devices in your lab.
- Isolate them. Put every management interface on a dedicated management network. Physical separation, with its own switch, is best. At minimum use a dedicated VLAN with firewall rules that allow access only from one administrative workstation or jump host. In our example network this is VLAN 10, 10.42.10.0/24.
- Never expose them to the internet. Reach them through a VPN or a hardened jump host, never through port forwarding. Our field note Keep Management Off the Internet explains why.
- Change default credentials and create a named administrative account instead of using the built-in one.
- Update the firmware and subscribe to whatever security notices the vendor publishes.
- Turn off what you do not use, such as legacy protocols and unencrypted interfaces.
Caution
Some controllers can fail over to a shared host network port if the dedicated port is unplugged. Check this setting. Otherwise, a management interface you believe is isolated can appear on your main network after someone tidies the cables.
A sensible setup for most labs
- Enable and update the management controller on every server that has one, and connect its dedicated port to the management network.
- Add one KVM-over-IP device and a small KVM switch for the machines that lack a controller.
- Keep a USB serial adapter in the rack for the firewall and switches.
- Reach all of it only through your VPN or a jump host, with multi-factor authentication where available.
- Write down every management IP address, console speed and the location of each credential in your lab documentation. See Documentation and Technical Debt.