In modern enterprise storage procurement, TrueNAS SCALE has emerged as the premier Linux-based OpenZFS platform for mid-market data storage, backup repositories, and hyperconverged SMB workloads. However, enterprise systems engineers migrating from legacy TrueNAS CORE (FreeBSD) frequently encounter an immediate architectural shock: TrueNAS SCALE’s default memory management behavior restricts the OpenZFS Adaptive Replacement Cache (ARC) to approximately 50% of physical system RAM.

For high-throughput environments running VMware datastores, PostgreSQL databases, or multi-tenant SMB file shares, leaving half of your expensive DDR5 ECC registered memory idle from a storage caching perspective severely degrades IOPS performance and inflates disk seek latency. Conversely, improperly tuning ARC limits on mixed virtualization nodes triggers Linux kernel memory pressure that can cause catastrophic process termination. This technical guide delivers an exhaustive forensic breakdown of ZFS ARC memory sizing, Linux kernel tuning, and storage procurement optimization.

What Is the Optimal ZFS ARC Size on TrueNAS SCALE?

The OpenZFS Adaptive Replacement Cache is widely recognized as the most sophisticated in-memory storage cache in enterprise computing. Unlike standard Least Recently Used (LRU) algorithms, ARC dynamically balances between Most Recently Used (MRU) and Most Frequently Used (MFU) data structures, ensuring that frequently queried database tables and hot files remain cached in lightning-fast DRAM.

In the FreeBSD-based TrueNAS CORE architecture, OpenZFS interacted directly with the BSD virtual memory manager, allowing ARC to consume nearly 90% of physical RAM and releasing memory instantaneously when user processes requested allocation. Under TrueNAS SCALE’s Debian Linux kernel, however, the kernel memory manager treats OpenZFS as an out-of-tree module. Historically, the Linux kernel is sluggish at reclaiming slab memory when applications demand sudden allocations. To prevent system deadlocks and kernel panics, OpenZFS on Linux defaults c_max (the maximum ARC size) to min(50% of system RAM, all RAM minus 1GB).

Kernel Tuning Mechanics: Configuring `zfs_arc_max`

To inspect your current ARC configuration and memory utilization in TrueNAS SCALE, administrators can run the following forensic commands from the system shell:

# View current ARC target size and current usage (in bytes)
arc_summary | grep -E "ARC size|Target size|Min size|Max size"

# Or query the kernel module parameter directly
cat /sys/module/zfs/parameters/zfs_arc_max

If the value returned by zfs_arc_max is 0, OpenZFS is operating in dynamic mode, capping ARC at the default 50% limit. To manually set ARC to a specific threshold—for instance, 96GB on a server with 128GB of RAM—calculate the value in bytes (96 * 1024 * 1024 * 1024 = 103079215104) and write it directly to the kernel parameter:

# Dynamically tune ARC max without rebooting
echo 103079215104 > /sys/module/zfs/parameters/zfs_arc_max

Because changes to /sys reset upon server reboot, TrueNAS SCALE administrators must persist this setting through the TrueNAS Web UI. Navigate to System Settings -> Advanced -> Init/Shutdown Scripts, create a new script with Type Command, set the When trigger to Post Init, and paste the echo [bytes] > /sys/module/zfs/parameters/zfs_arc_max command. This ensures the engineered memory boundary survives system updates and reboots.

Preventing Linux Out-Of-Memory (OOM) Killer Disasters

While maximizing ARC allocation provides dramatic speed benefits for pure file servers, it introduces existential stability risks in hyperconverged environments. When a KVM virtual machine or Docker container requests memory, the Linux kernel must allocate pages instantly. If ARC has consumed 85% of physical memory, the kernel signals OpenZFS to prune ARC pages. However, freeing millions of cached slab objects from OpenZFS takes tens or hundreds of milliseconds.

If the Linux kernel exhausts unreserved pages before OpenZFS can shrink ARC, the kernel’s emergency protection mechanism—the Out-Of-Memory (OOM) Killer—triggers immediately. The OOM killer evaluates running processes using an internal badness heuristic (/proc/[pid]/oom_score). Virtual machine hypervisor processes (qemu-system-x86_64) invariably register the highest badness scores because they hold massive contiguous blocks of anonymous memory. As a result, the Linux kernel terminates your mission-critical production virtual machines in a fraction of a millisecond to protect system stability.

To insulate enterprise nodes from OOM catastrophes, administrators hosting VMs on TrueNAS SCALE must calculate strict memory budgets: Total RAM - (VM Allocated RAM + Container RAM + 16GB Base System Reserve) = Maximum Safe ARC Limit. Never allow the sum of configured virtual machine memory and zfs_arc_max to exceed 85% of total system memory.

DDR5 ECC RAM Procurement vs. NVMe L2ARC Economics

When storage arrays suffer from low ARC hit ratios (e.g., below 85%), procurement officers frequently ask whether it is more cost-effective to upgrade server DRAM or deploy enterprise NVMe SSDs as a Second-Level Adaptive Replacement Cache (L2ARC).

From a technical standpoint, physical DRAM operates at latencies under 100 nanoseconds with throughput exceeding 200 GB/s across multi-channel DDR5 buses. Enterprise NVMe SSDs (such as Kioxia CM7 or Solidigm D7-P5520) operate at latencies between 10 to 50 microseconds (100x to 500x slower than DRAM) with throughput peaking around 7 to 14 GB/s. Crucially, every byte stored in L2ARC consumes memory headers in primary ARC (approximately 50 to 80 bytes of RAM per L2ARC block). Adding a massive 2TB L2ARC SSD can consume several gigabytes of precious DRAM just to maintain index tables.

Therefore, enterprise procurement should follow a strict hierarchy: Maximize motherboard DRAM first. Purchasing high-density 64GB or 96GB DDR5-5600 ECC RDIMM modules provides unmatched IOPS acceleration. Only when all physical DIMM slots on the server motherboard are fully populated should enterprise engineers procure high-endurance NVMe drives for L2ARC to hold secondary overflow datasets.