U3 Storage and Data

ZFS and filesystems: integrity, layouts and myths

A filesystem with checksums can tell you when your data has silently changed and, with redundancy, fix it. That ability comes at the price of planning. This guide explains the planning in plain terms and clears up the myths that scare people away.

At a glance

  • ZFS combines volume management and the filesystem, so it checksums every block and can repair corruption from redundant copies.
  • Design the layout before you create the pool. Groups of drives, called vdevs, are hard to change later.
  • Choose two of capacity efficiency, performance and easy expansion. Mirrors favor the last two; wide parity groups favor the first.
  • Schedule scrubs, snapshots and replication from day one. They are why you chose ZFS.
  • ECC memory is good practice, not a requirement. There is no fixed rule for RAM per terabyte.

Why checksums matter

Traditional setups stack separate layers: software RAID manages disks, and a filesystem such as ext4 sits on top. The RAID layer cannot tell which copy of a block is correct when two disagree, and the filesystem cannot see the disks at all.

ZFS and Btrfs are copy-on-write filesystems that store a checksum for every block. When data is read, the checksum is verified. If it does not match and a redundant copy exists, the filesystem returns the good copy and repairs the bad one. A scrub reads every block on purpose to find and fix problems before you need the data.

Myth

"I have never had bit rot." Without checksums, you would not know. Silent corruption is usually discovered years later, when a photo opens half grey and every snapshot that still held a good copy has long since expired.

How a ZFS pool is built

The vocabulary is the hardest part. Learn these four terms and the rest follows.

TermWhat it is
DriveA physical disk.
VdevA group of drives with its own redundancy: a mirror, a parity group (RAIDZ1, RAIDZ2 or RAIDZ3), or a single disk.
PoolOne or more vdevs combined into one storage space. Data is spread across all vdevs.
Dataset or zvolA filesystem (dataset) or block device (zvol) carved from the pool, each with its own settings, snapshots and quotas.

Warning

Redundancy lives inside each vdev, not across the pool. If any single vdev fails completely, the entire pool is lost. Never add a single unprotected disk to a pool that otherwise uses mirrors or parity.

Mirrors or parity

LayoutSurvivesUsable spaceCharacter
Two-way mirrors1 drive per mirror50 percentFast random I/O, fast rebuilds, expand two drives at a time
RAIDZ1 (single parity)1 drive per vdev(n-1)/nEfficient, but a long rebuild with no remaining protection on large drives
RAIDZ2 (double parity)2 drives per vdev(n-2)/nThe usual choice for bulk storage with large drives
RAIDZ3 (triple parity)3 drives per vdev(n-3)/nVery wide vdevs and archives

Each vdev performs roughly like one drive for random I/O, so more vdevs mean more performance. Ten drives arranged as two five-wide RAIDZ2 vdevs are faster than one ten-wide RAIDZ2 vdev, survive two failures in each half and rebuild sooner, at the cost of two more drives' worth of parity. Keep vdevs in a pool the same type and width. Very wide parity groups, beyond about 10 to 12 drives, rebuild slowly and stay degraded for longer.

Rule of thumb

For virtual machine storage, use mirrors on SSDs. For bulk media and archives on large hard drives, use RAIDZ2 vdevs of six to ten drives. Use RAIDZ1 only with small drives or for data that is fully backed up elsewhere.

Growing a pool

  • Add a vdev. Add another group of the same type and width. A pool of one six-drive RAIDZ2 vdev grows by six more drives at a time.
  • Replace drives with larger ones. Replace every drive in a vdev, one at a time with a resilver after each, and the extra capacity appears after the last one.
  • Widen a parity group. Recent releases can add a single drive to an existing RAIDZ vdev. Data written before the expansion keeps its original ratio of data to parity until it is rewritten, so the gain is smaller than a freshly built vdev of the same width would give.

Plan as if a pool can never shrink. Size the chassis for the drives you will add in three to four years, and see the Storage Capacity Calculator to compare layouts.

Settings that are hard to change later

  • Sector alignment. Create vdevs with 4K sector alignment (ashift of 12) for modern drives. It is fixed at creation.
  • Encryption. Decide on native encryption when creating a dataset. Keep the key or passphrase in at least two places that are not the NAS itself.
  • Dataset structure. Plan the tree first: separate datasets for documents, photos, media, backups and VM storage, so each can have its own snapshots and settings.

Settings that are easy and worth changing

  • Compression on. Lightweight compression costs little CPU and often improves performance by reducing disk I/O.
  • Access time updates off for most datasets, to avoid a write for every read.
  • Record size by workload. Large records such as 1 MB suit big media files; smaller records suit databases and VM disks.

Snapshots, scrubs and replication

A snapshot records the state of a dataset instantly and costs only the space of blocks that change afterward. If 100 MB changes each day, two weeks of daily snapshots use about 1.4 GB. You can roll back a dataset, browse a snapshot to recover a single file, or clone a snapshot into a new dataset to investigate without touching live data.

TaskSuggested schedule
Snapshots of documents and photosHourly, kept for 2 days; daily, kept for 30 days; weekly, kept for 6 months
Snapshots of VM storageDaily, kept for 14 days
ScrubMonthly for hard drive pools, more often for small SSD pools if you like
Drive self-testsShort tests weekly, long tests monthly, staggered away from scrubs
Replication to a second systemDaily, or after each important snapshot

Avoid letting snapshots accumulate into the tens of thousands, which slows listing and management. Expire them on a schedule.

Replication sends snapshots to another ZFS system at the block level, so only changed blocks travel. Encrypted datasets can be sent in raw form, which means the receiving system stores your data without ever being able to read it. That makes a friend's NAS a safe off-site target. Over the internet, the link is the bottleneck, so encrypting the transport with SSH costs nothing noticeable.

Caution

Snapshots are not backups. They live in the same pool as the data, and anyone with administrative access can delete them. Treat them as an undo button, and keep real backups on separate systems with separate credentials. See Backup Strategy.

Myths and facts

ClaimReality
ZFS requires ECC memoryECC is good practice for any server holding important data. ZFS without ECC is still safer than a filesystem without checksums.
You need 1 GB of RAM per TB of storageThere is no such requirement for ordinary use. Extra RAM improves caching. Deduplication is the exception and is rarely worth it at home.
A cache SSD makes everything fasterA read cache device helps few home workloads. Add RAM first.
A separate log device speeds up all writesIt only helps synchronous writes, such as NFS for VM storage or databases. Use a device with power-loss protection.
High memory use means something is wrongZFS uses free memory as cache and releases it under pressure. On a hypervisor you may want to cap the cache.

One more caution on special devices: a metadata vdev on SSDs can speed up pools with many small files, but if it is lost the whole pool is lost. Always mirror it.

Updates and feature flags

New filesystem features arrive in upstream releases first and reach stable appliance releases later. For a pool that holds important data, stay on the stable release of your NAS distribution. After a major operating system update, the system may offer to upgrade the pool's feature flags. Wait until you are sure you will not roll back, because an older system cannot import a pool with newer features enabled. Platforms with boot environments let you roll back the operating system itself, which makes updates far less stressful.

Other filesystems and where they fit

Btrfs

Btrfs is a Linux-native copy-on-write filesystem with checksums, snapshots and subvolumes. Its snapshot integration is excellent, and several distributions take automatic snapshots before updates so you can boot the previous state. Its mirroring modes are widely used, but its parity RAID modes are still not recommended for important data; many turnkey NAS products layer Btrfs over Linux software RAID instead. Use Btrfs's own usage commands to check free space, because general tools can report space that does not really exist once snapshots and metadata are counted.

ext4 and XFS

For general Linux servers and virtual machines, ext4 and XFS are both mature, fast and dependable. For almost every homelab workload the choice is a coin flip, so use your distribution's default. XFS cannot be shrunk; ext4 can, but shrinking is risky.

LVM

The logical volume manager sits under ext4 or XFS and lets you grow volumes online and take temporary snapshots. Leave 10 to 20 percent of the volume group unallocated for emergencies and snapshots. Remove snapshots as soon as you are done with them, because a snapshot that fills up becomes invalid. Never span a volume group across several disks without RAID underneath; losing one disk would lose everything.

From the Bench

We keep one rule above all others: no filesystem is a substitute for backups. Every filesystem on the Bench has been misconfigured by one of us at least once. The data survived because it existed somewhere else.