U3 Storage and Data
ZFS and filesystems: integrity, layouts and myths
A filesystem with checksums can tell you when your data has silently changed and, with redundancy, fix it. That ability comes at the price of planning. This guide explains the planning in plain terms and clears up the myths that scare people away.
At a glance
- ZFS combines volume management and the filesystem, so it checksums every block and can repair corruption from redundant copies.
- Design the layout before you create the pool. Groups of drives, called vdevs, are hard to change later.
- Choose two of capacity efficiency, performance and easy expansion. Mirrors favor the last two; wide parity groups favor the first.
- Schedule scrubs, snapshots and replication from day one. They are why you chose ZFS.
- ECC memory is good practice, not a requirement. There is no fixed rule for RAM per terabyte.
Why checksums matter
Traditional setups stack separate layers: software RAID manages disks, and a filesystem such as ext4 sits on top. The RAID layer cannot tell which copy of a block is correct when two disagree, and the filesystem cannot see the disks at all.
ZFS and Btrfs are copy-on-write filesystems that store a checksum for every block. When data is read, the checksum is verified. If it does not match and a redundant copy exists, the filesystem returns the good copy and repairs the bad one. A scrub reads every block on purpose to find and fix problems before you need the data.
Myth
"I have never had bit rot." Without checksums, you would not know. Silent corruption is usually discovered years later, when a photo opens half grey and every snapshot that still held a good copy has long since expired.
How a ZFS pool is built
The vocabulary is the hardest part. Learn these four terms and the rest follows.
| Term | What it is |
|---|---|
| Drive | A physical disk. |
| Vdev | A group of drives with its own redundancy: a mirror, a parity group (RAIDZ1, RAIDZ2 or RAIDZ3), or a single disk. |
| Pool | One or more vdevs combined into one storage space. Data is spread across all vdevs. |
| Dataset or zvol | A filesystem (dataset) or block device (zvol) carved from the pool, each with its own settings, snapshots and quotas. |
Warning
Redundancy lives inside each vdev, not across the pool. If any single vdev fails completely, the entire pool is lost. Never add a single unprotected disk to a pool that otherwise uses mirrors or parity.
Mirrors or parity
| Layout | Survives | Usable space | Character |
|---|---|---|---|
| Two-way mirrors | 1 drive per mirror | 50 percent | Fast random I/O, fast rebuilds, expand two drives at a time |
| RAIDZ1 (single parity) | 1 drive per vdev | (n-1)/n | Efficient, but a long rebuild with no remaining protection on large drives |
| RAIDZ2 (double parity) | 2 drives per vdev | (n-2)/n | The usual choice for bulk storage with large drives |
| RAIDZ3 (triple parity) | 3 drives per vdev | (n-3)/n | Very wide vdevs and archives |
Each vdev performs roughly like one drive for random I/O, so more vdevs mean more performance. Ten drives arranged as two five-wide RAIDZ2 vdevs are faster than one ten-wide RAIDZ2 vdev, survive two failures in each half and rebuild sooner, at the cost of two more drives' worth of parity. Keep vdevs in a pool the same type and width. Very wide parity groups, beyond about 10 to 12 drives, rebuild slowly and stay degraded for longer.
Rule of thumb
For virtual machine storage, use mirrors on SSDs. For bulk media and archives on large hard drives, use RAIDZ2 vdevs of six to ten drives. Use RAIDZ1 only with small drives or for data that is fully backed up elsewhere.
Growing a pool
- Add a vdev. Add another group of the same type and width. A pool of one six-drive RAIDZ2 vdev grows by six more drives at a time.
- Replace drives with larger ones. Replace every drive in a vdev, one at a time with a resilver after each, and the extra capacity appears after the last one.
- Widen a parity group. Recent releases can add a single drive to an existing RAIDZ vdev. Data written before the expansion keeps its original ratio of data to parity until it is rewritten, so the gain is smaller than a freshly built vdev of the same width would give.
Plan as if a pool can never shrink. Size the chassis for the drives you will add in three to four years, and see the Storage Capacity Calculator to compare layouts.
Settings that are hard to change later
- Sector alignment. Create vdevs with 4K sector alignment (ashift of 12) for modern drives. It is fixed at creation.
- Encryption. Decide on native encryption when creating a dataset. Keep the key or passphrase in at least two places that are not the NAS itself.
- Dataset structure. Plan the tree first: separate datasets for documents, photos, media, backups and VM storage, so each can have its own snapshots and settings.
Settings that are easy and worth changing
- Compression on. Lightweight compression costs little CPU and often improves performance by reducing disk I/O.
- Access time updates off for most datasets, to avoid a write for every read.
- Record size by workload. Large records such as 1 MB suit big media files; smaller records suit databases and VM disks.
Snapshots, scrubs and replication
A snapshot records the state of a dataset instantly and costs only the space of blocks that change afterward. If 100 MB changes each day, two weeks of daily snapshots use about 1.4 GB. You can roll back a dataset, browse a snapshot to recover a single file, or clone a snapshot into a new dataset to investigate without touching live data.
| Task | Suggested schedule |
|---|---|
| Snapshots of documents and photos | Hourly, kept for 2 days; daily, kept for 30 days; weekly, kept for 6 months |
| Snapshots of VM storage | Daily, kept for 14 days |
| Scrub | Monthly for hard drive pools, more often for small SSD pools if you like |
| Drive self-tests | Short tests weekly, long tests monthly, staggered away from scrubs |
| Replication to a second system | Daily, or after each important snapshot |
Avoid letting snapshots accumulate into the tens of thousands, which slows listing and management. Expire them on a schedule.
Replication sends snapshots to another ZFS system at the block level, so only changed blocks travel. Encrypted datasets can be sent in raw form, which means the receiving system stores your data without ever being able to read it. That makes a friend's NAS a safe off-site target. Over the internet, the link is the bottleneck, so encrypting the transport with SSH costs nothing noticeable.
Caution
Snapshots are not backups. They live in the same pool as the data, and anyone with administrative access can delete them. Treat them as an undo button, and keep real backups on separate systems with separate credentials. See Backup Strategy.
Myths and facts
| Claim | Reality |
|---|---|
| ZFS requires ECC memory | ECC is good practice for any server holding important data. ZFS without ECC is still safer than a filesystem without checksums. |
| You need 1 GB of RAM per TB of storage | There is no such requirement for ordinary use. Extra RAM improves caching. Deduplication is the exception and is rarely worth it at home. |
| A cache SSD makes everything faster | A read cache device helps few home workloads. Add RAM first. |
| A separate log device speeds up all writes | It only helps synchronous writes, such as NFS for VM storage or databases. Use a device with power-loss protection. |
| High memory use means something is wrong | ZFS uses free memory as cache and releases it under pressure. On a hypervisor you may want to cap the cache. |
One more caution on special devices: a metadata vdev on SSDs can speed up pools with many small files, but if it is lost the whole pool is lost. Always mirror it.
Updates and feature flags
New filesystem features arrive in upstream releases first and reach stable appliance releases later. For a pool that holds important data, stay on the stable release of your NAS distribution. After a major operating system update, the system may offer to upgrade the pool's feature flags. Wait until you are sure you will not roll back, because an older system cannot import a pool with newer features enabled. Platforms with boot environments let you roll back the operating system itself, which makes updates far less stressful.
Other filesystems and where they fit
Btrfs
Btrfs is a Linux-native copy-on-write filesystem with checksums, snapshots and subvolumes. Its snapshot integration is excellent, and several distributions take automatic snapshots before updates so you can boot the previous state. Its mirroring modes are widely used, but its parity RAID modes are still not recommended for important data; many turnkey NAS products layer Btrfs over Linux software RAID instead. Use Btrfs's own usage commands to check free space, because general tools can report space that does not really exist once snapshots and metadata are counted.
ext4 and XFS
For general Linux servers and virtual machines, ext4 and XFS are both mature, fast and dependable. For almost every homelab workload the choice is a coin flip, so use your distribution's default. XFS cannot be shrunk; ext4 can, but shrinking is risky.
LVM
The logical volume manager sits under ext4 or XFS and lets you grow volumes online and take temporary snapshots. Leave 10 to 20 percent of the volume group unallocated for emergencies and snapshots. Remove snapshots as soon as you are done with them, because a snapshot that fills up becomes invalid. Never span a volume group across several disks without RAID underneath; losing one disk would lose everything.
From the Bench
We keep one rule above all others: no filesystem is a substitute for backups. Every filesystem on the Bench has been misconfigured by one of us at least once. The data survived because it existed somewhere else.