- The Double-Write Penalty: Running Ceph Object Storage Daemons (OSDs) on top of ZFS datasets forces every I/O to traverse two independent copy-on-write (CoW) journaling pipelines—Ceph BlueStore RocksDB write-ahead logs and the ZFS Intent Log (ZIL)—causing up to an 82% drop in 4K random write IOPS.
- BlueStore Direct Raw Access: Ceph BlueStore was engineered specifically to consume raw, unformatted block devices directly without an underlying filesystem. Bypassing ZFS yields native PCIe bus throughput and predictable sub-millisecond latencies.
- Correct Architecture Separation: Use ZFS for ultra-fast, local NVMe/SSD single-node storage pools, and deploy Ceph directly on dedicated raw NVMe drives across 3 or more nodes connected via dedicated 10GbE or 25GbE mesh networking.
One of the most frequent architectural queries emerging in homelab and enterprise Proxmox communities is whether administrators can layer Ceph storage pools directly onto an existing ZFS pool. The impulse is understandable: administrators want the robust local data integrity, snapshots, and ARC caching of ZFS combined with the automated distributed replication and high-availability failover of Ceph.
However, from a storage engineering perspective, running Ceph on top of ZFS is a severe antipattern that degrades storage performance, balloons write amplification, and introduces catastrophic latency cascades during cluster scrub operations. Understanding why requires dissecting Ceph’s BlueStore storage engine.
Architectural Breakdown: Ceph BlueStore vs. ZFS CoW Stacking
| Architectural Dimension | Ceph on Raw Block Devices (Native BlueStore) | Ceph Layered on ZFS (Antipattern) |
|---|---|---|
| Filesystem Overhead | Zero. Direct block I/O via custom BlueFS and RocksDB. | Extreme. BlueStore writes to virtual block device traversing ZFS POSIX layer. |
| Write Amplification Factor (WAF) | 1.2x – 1.8x (Ceph internal replication and metadata updates). | 4.5x – 9.0x (Combined Ceph WAL + ZFS ZIL + Copy-on-Write fragmentation). |
| 4K Random Write IOPS | Native hardware speed (50,000 – 150,000 IOPS per enterprise NVMe). | Severely degraded (often sub-8,000 IOPS due to double synchronization). |
| Memory Consumption | Predictable: 1GB – 4GB RAM per OSD targeted via OSD memory target. | Uncontrolled: Ceph OSD daemon memory fights with ZFS ARC for host RAM. |
| Scrubbing & Deep Scrub Latency | Handled entirely within Ceph CRUSH algorithm without host locks. | Massive I/O stalls when ZFS scrub overlaps with Ceph PG scrub. |
Why BlueStore Made Filesystem Layering Obsolete
In older Ceph versions (FileStore era, pre-Luminous), Ceph actually relied on a host POSIX filesystem (primarily XFS or Btrfs) to store object data. This required POSIX journal commits that crippled performance. To eliminate this bottleneck, the Ceph core team engineered BlueStore.
BlueStore operates without any operating system filesystem. It claims raw block partitions directly, managing raw data blocks using its own lightweight allocator and storing object metadata in an embedded RocksDB key-value database running atop BlueFS. When you place a BlueStore OSD inside a ZFS zvol or directory, you force BlueStore’s direct block calls through ZFS transaction groups (TXGs), causing double journaling, double checksumming, and severe latency spikes.
If you are planning high-density cluster storage across low-power hardware, check out our production teardown of The 3-Node Proxmox Micro-Cluster with Ceph NVMe Mesh.
The Production Blueprint: How to Properly Architect Proxmox Storage
To extract maximum performance from both technologies in Proxmox VE 8.x, adhere strictly to a separated role architecture:
- Use ZFS Exclusively for Single-Node Storage: If you need screaming-fast local storage for databases, compilation workloads, or standalone VMs, configure mirrored NVMe drives as a native ZFS pool (zfs-local). Take advantage of ZFS snapshots and asynchronous replication via `pvesr`.
- Dedicate Raw NVMe Disks to Ceph: For VMs that require automated high availability (HA) and live migration without storage downtime, pass dedicated, unformatted NVMe drives directly to Ceph OSDs. Never format the drive with EXT4, XFS, or ZFS before provisioning an OSD.
- Isolate Network Fabrics: Ceph requires low-latency networking. Ensure Ceph public and cluster traffic runs on a dedicated, isolated 10GbE, 25GbE, or full-mesh network interface, separate from VM and Corosync traffic.
For homelabs demanding extreme block storage throughput without Ceph clustering overhead, explore our guide on NVMe-oF & ZFS dRAID 100GbE All-Flash SANs.
Frequently Asked Questions (PAA Direct Answers)
Technically yes via the CLI, but it is strictly advised against in production and homelab environments. The zvol adds a layer of copy-on-write abstraction that destroys random write performance and can cause OSD heartbeat timeouts during heavy storage load, leading to cluster instability and false-positive OSD failure states.
ZFS is exponentially faster on a single node. Ceph introduces network socket latency, hashing via the CRUSH map, and distributed consensus overhead that yields zero benefits on a single machine. Ceph only becomes advantageous when spanning across 3 or more physical nodes for high availability.
By default, Proxmox allocates an `osd_memory_target` of approximately 4GB of RAM per OSD. In low-power homelabs with constrained memory, this can be safely reduced to 2GB or 2.5GB for NVMe drives using Ceph configuration keys, but setting it below 1.5GB risks OSD crashes during PG peering and backfilling.
Never compromise your storage stack by forcing Ceph to run on top of ZFS. If your cluster demands automated VM failover and seamless shared storage across multiple physical machines, give Ceph raw, unpartitioned NVMe block devices to let BlueStore execute at native bus speeds. If you are building a high-performance compute node with local storage, let ZFS manage your drives natively. Both tools are masters of their domain—keep them cleanly separated.