ClickHouse Disaster Recovery with ZFS Snapshots and zrepl

ClickHouse is fantastic at storing huge volumes of data cheaply. It is less opinionated about what happens when the server underneath it catches fire. If you run ClickHouse on your own hardware or VMs, disaster recovery is your problem to solve — and one of the cleanest ways to solve it is to put ClickHouse's data on a ZFS dataset and replicate that dataset to another machine with zrepl.

This post walks through the whole thing: why ZFS snapshots are a surprisingly good fit for ClickHouse, how to set up continuous off-box replication, how to actually restore after a failure, and the consistency caveats you need to understand before you trust it in production.

Why ZFS snapshots work well for ClickHouse

The instinct with most databases is "you can't just snapshot the files — you'll get a torn, inconsistent copy." For ClickHouse's MergeTree family, that instinct is mostly wrong, and it's worth understanding why.

MergeTree stores data as immutable parts. When you insert data, ClickHouse writes a new part on disk. It never edits an existing part in place. Background merges combine small parts into bigger ones — but a merge writes a brand-new part and only then drops the old ones. Mutations (ALTER ... UPDATE/DELETE) work the same way: they produce new parts rather than rewriting bytes inside existing ones.

That immutability is the key. A ZFS snapshot is atomic — it captures the exact state of the entire dataset at a single instant, with no half-written blocks. Combine those two facts and you get the important property: a ZFS snapshot of a ClickHouse data directory captures a set of complete, self-consistent, immutable parts. Nothing is half-modified, because ClickHouse never modifies anything half-way.

There's a second, underrated benefit. Because a ZFS snapshot covers the whole dataset at once, it's atomic across all your tables and databases simultaneously. Logical backup tools often struggle to give you a single point-in-time across many tables; a filesystem snapshot gives it to you for free.

This makes ZFS + zrepl attractive as a disaster-recovery layer:

• Continuous, incremental replication to a remote server with minimal bandwidth (only changed blocks travel).
• Point-in-time recovery from any retained snapshot.
• Fast restores — you roll back or receive a snapshot; there's no re-inserting or re-merging of data.

It is not a replacement for everything (more on the caveats later), but as an operational safety net it's hard to beat.

What is zrepl?

zrepl is an integrated tool for ZFS replication. You describe your intent in a YAML config — which datasets to snapshot, how often, where to send them, and how long to keep them — and a daemon on each host takes care of snapshotting, incremental zfs send/zfs receive, and pruning old snapshots on a schedule.

The model is a pair of jobs:

• A push job on the production server: it creates snapshots, keeps a short local history so incremental sends are possible, connects to the backup server, and prunes.
• A sink job on the backup server: it listens for the push job and writes the received datasets under a destination dataset.

(zrepl also supports a pull model, where the backup server initiates the connection. Push is simpler to reason about for a single prod → backup relationship, so that's what we'll use.)

Prerequisites

• Two servers: Source (runs ClickHouse, data on a ZFS dataset) and Target (receives replicas).
• ZFS installed on both, with a pool on each. In the examples below:
  • Source pool data, with ClickHouse data on data/clickhouse.
  • Target pool backup.
• Network reachability from Source to Target on the port zrepl will use (we'll use 8888).
• Root/sudo on both hosts.

If ClickHouse is already running on a non-ZFS disk, you'll need to migrate its data directory (/var/lib/clickhouse) onto a ZFS dataset first. That migration is out of scope here, but the short version is: stop ClickHouse, create the dataset, rsync the data over, point ClickHouse at it (or mount the dataset at the data path), and start it again.

Step 1 — Install zrepl on both servers

zrepl publishes an APT repository. On both the source and target, add the repo and install the package. Save this as install-zrepl.sh and run it:

#!/usr/bin/env bash
set -euo pipefail
 
KEY_URL="https://zrepl.cschwarz.com/apt/apt-key.asc"
KEYRING="/usr/share/keyrings/zrepl.gpg"
REPO_FILE="/etc/apt/sources.list.d/zrepl.list"
 
sudo apt update
sudo apt install -y curl gnupg lsb-release
 
# Import the signing key
curl -fsSL "$KEY_URL" | gpg --dearmor | sudo tee "$KEYRING" > /dev/null
 
# Register the repository for this distro + codename
ARCH="$(dpkg --print-architecture)"
DISTRO="$(lsb_release -is | tr '[:upper:]' '[:lower:]')"
CODENAME="$(lsb_release -cs | tr '[:upper:]' '[:lower:]')"
echo "deb [arch=$ARCH signed-by=$KEYRING] https://zrepl.cschwarz.com/apt/${DISTRO} ${CODENAME} main" \
  | sudo tee "$REPO_FILE" > /dev/null
 
sudo apt update
sudo apt install -y zrepl
 
# Pin the version so an unattended upgrade can't surprise you
sudo apt-mark hold zrepl

Then enable and start the service on both hosts:

sudo systemctl enable --now zrepl
zrepl version
systemctl status zrepl.service

On the target, create the dataset that will hold the received data:

zfs create backup/clickhouse-dr

If you run a host firewall on the target, allow the zrepl port:

sudo ufw allow 8888/tcp

The scenario we're building

To make the config concrete, here's the policy we'll implement:

• Snapshot the ClickHouse dataset on the source every 10 minutes.
• Incrementally replicate each new snapshot to the target.
• On the source, keep only a handful of recent snapshots (just enough for incremental sends) to conserve disk.
• On the target, keep a fading history: recent snapshots at fine granularity, then hourly, then daily, so you can recover to many different points in time without storing everything forever.

Step 2 — Configure the source (push job)

zrepl's config lives at /etc/zrepl/zrepl.yml. On the source:

global:
  logging:
    # Log to syslog so journald stays happy
    - type: syslog
      format: human
      level: warn
 
jobs:
  - name: clickhouse_to_backup
    type: push
    connect:
      type: tcp
      address: "TARGET_IP:8888"       # the target server's address
    filesystems:
      "data/clickhouse": true          # the dataset holding ClickHouse data
    snapshotting:
      type: periodic
      interval: 10m
      prefix: "zrepl_"                 # tag our snapshots so pruning only touches ours
    pruning:
      # On the SOURCE: keep just enough for incremental replication
      keep_sender:
        - type: not_replicated         # never delete a snapshot that hasn't shipped yet
        - type: last_n
          count: 5
      # On the TARGET: keep a fading history
      keep_receiver:
        - type: grid
          grid: 1x1h(keep=all) | 24x1h | 30x1d
          regex: "^zrepl_"

A few things worth understanding here:

• not_replicated on the sender is the safety rule that matters most: it guarantees a snapshot is never pruned locally before it has been successfully replicated to the target.
• The grid on the receiver reads left to right: keep everything from the last hour, then thin to 24 hourly snapshots, then 30 daily ones. Tune these numbers to your recovery-window and storage budget.
• The prefix plus the pruning regex ensures zrepl only ever prunes snapshots it created — any manual snapshots you take are left alone.

Apply it:

sudo systemctl restart zrepl.service

Step 3 — Configure the target (sink job)

On the target, /etc/zrepl/zrepl.yml:

global:
  logging:
    - type: syslog
      format: human
      level: warn
 
jobs:
  - name: receive_clickhouse
    type: sink
    serve:
      type: tcp
      listen: ":8888"
      clients:
        "SOURCE_IP": "clickhouse"      # client identity -> subtree name
    root_fs: "backup/clickhouse-dr"    # received datasets land under here

Restart:

sudo systemctl restart zrepl.service

Received data will appear under backup/clickhouse-dr/clickhouse/..., following the root_fs + client-identity layout.

A note on production: use TLS, not plain TCP

The config above uses type: tcp, which is fine on a trusted private network but sends your data unencrypted. For anything crossing a network you don't fully control, switch both sides to type: tls. zrepl uses mutual TLS with self-signed certificates: you generate a cert/key pair per host, each side trusts the other's certificate, and the client identity comes from the certificate's common name. It's a few extra lines in connect/serve and well worth it for a DR link.

Step 4 — Verify replication

Give it a snapshot interval or two, then check that snapshots exist and are landing on the target.

On the source, confirm snapshots are being created:

zfs list -t snapshot -o name,creation | grep clickhouse

On the target, confirm they're being received:

zfs list -t snapshot -o name,creation | grep clickhouse

You should see the same zrepl_-prefixed snapshot names showing up on the target shortly after they appear on the source. You can also watch a replication run live from the source:

zrepl status

If snapshots appear on the source but never on the target, the usual suspects are: the target's 8888 port not reachable (firewall), a mismatch between the source IP and the clients entry in the sink config, or the client identity not being a valid ZFS path component.

Step 5 — Disaster recovery: restoring ClickHouse

When the source fails, how you recover depends on where a good copy lives.

Always stop ClickHouse before restoring its data files. Restoring underneath a running server will corrupt its view of the world:

sudo systemctl stop clickhouse-server.service

Case A — Rolling back locally

If the data is intact locally and you just need to return to an earlier point (say a bad bulk operation), roll the dataset back to a snapshot:

# List available snapshots and pick one
zfs list -t snapshot data/clickhouse
 
# Roll back (this discards changes made after the snapshot)
sudo zfs rollback data/clickhouse@zrepl_<timestamp>

Then start ClickHouse again and verify.

Case B — Restoring from the target server

If the source is gone or its disk is toast, pull a snapshot back from the target. Send it over SSH and receive it into a dataset on the recovered source:

# On the recovered source: receive a snapshot streamed from the target
ssh TARGET_IP "zfs send backup/clickhouse-dr/clickhouse@zrepl_<timestamp>" \
  | zfs receive -F data/clickhouse

If you received into a temporary dataset for safety, promote it into place by rolling the live dataset to the received snapshot:

sudo zfs rollback data/clickhouse@zrepl_<timestamp>

Start ClickHouse and validate:

sudo systemctl start clickhouse-server.service

A quick sanity check is to query a table that carries a timestamp and confirm the newest data matches the snapshot you restored:

SELECT max(event_time) FROM your_table;

For this to work in a real incident, make sure SSH access between the two hosts is set up before you need it. Recovery day is not the time to be exchanging keys.

Bonus — Expanding a ZFS disk when the backup fills up

The target holds a wider range of snapshots than the source, so it tends to run out of space first. If you've attached a larger disk (or grown the underlying virtual disk), tell ZFS to use the new space:

# Identify the device
lsblk
 
# Expand the pool onto the full device
sudo zpool online -e backup <device>

The -e flag expands the device to use all available space. On mirrors or raidz, every device in the group must be expanded before the new capacity becomes usable.

Consistency caveats you should actually know

This approach is solid, but "crash-consistent" is not the same as "transactionally perfect." A few honest caveats:

1. A snapshot is crash-consistent, like pulling the plug. If an insert is in flight at the exact instant of the snapshot, that partial part is simply absent (or discarded on startup). ClickHouse handles this gracefully on restart, but you may lose the last few seconds of un-flushed inserts. For most analytics workloads that's acceptable; if it isn't, pair this with a durable ingestion buffer (e.g., Kafka) so you can replay.

2. Replicated tables need Keeper too. If you use ReplicatedMergeTree, the source of truth for replication state lives in ClickHouse Keeper, not in the table's data directory. A ZFS snapshot of the data doesn't capture Keeper's metadata. For a full replicated-cluster DR story you also need a Keeper backup strategy, and on restore you may need SYSTEM RESTORE REPLICA to reconcile a restored replica with Keeper.

3. Put ClickHouse data on its own dataset. Snapshot the dataset that contains the ClickHouse store and nothing unrelated. Mixing other workloads into the same dataset muddies your recovery point and bloats your sends.

4. Test your restores. An untested backup is a rumor. Periodically receive a snapshot onto a spare box, start ClickHouse against it, and run a few queries. The confidence is worth the effort.

Where this fits in your backup strategy

ZFS + zrepl gives you a low-overhead, continuously-updated physical copy of your data on a second machine, with point-in-time snapshots and fast restores. That covers the "server died" and "someone ran a destructive command" scenarios extremely well.

For belt-and-suspenders coverage, combine it with a logical backup to object storage (for example, ClickHouse's native BACKUP command or a tool that uses ALTER TABLE ... FREEZE). Physical replication protects against hardware and operational failure; an off-site logical backup protects against pool corruption and gives you a portable, engine-level copy. Different failure modes, different tools — and together they're a genuinely resilient setup.

Start with the ZFS replica for your fast-recovery path, add Keeper backups if you run replicated tables, and keep a periodic logical backup off-site. That's a disaster-recovery posture you can actually sleep behind.