Storage disaggregation has reached an inflection point in enterprise virtualization and high-density home lab clusters. For decades, legacy SAN architectures relied on iSCSI over TCP or proprietary Fibre Channel fabrics. However, as Gen4 and Gen5 enterprise U.2/U.3 and E1.S NVMe SSDs capable of delivering 1,000,000+ random IOPS and sub-20 microsecond read latencies became standard, the legacy SCSI block stack (with its coarse SCSI commands, single command queue depth bottleneck of 32 commands, and heavy kernel context switching) emerged as the single biggest performance anchor in virtualized hypervisors.

In 2026, NVMe-over-TCP (NVMe-oF/TCP) has become the open-standard disaggregated storage protocol of choice for Proxmox VE 8.x and 9.x clusters. Unlike NVMe-over-RDMA (RoCEv2 or InfiniBand), which requires specialized lossless network switching, Priority Flow Control (PFC), and Explicit Congestion Notification (ECN) tuning, NVMe-over-TCP runs over ubiquitous, lossy commodity Ethernet switches. When engineered with line-rate 25GbE or 100GbE NICs and paired with an optimized storage target, NVMe-over-TCP delivers 92% to 96% of bare-metal NVMe performance at sub-15 microsecond latency overhead.

Architectural Comparison: iSCSI vs. NVMe-oF/RDMA vs. NVMe-oF/TCP

The performance differential between legacy block storage and modern fabric protocols comes down to queue depth parallelism and CPU interrupt processing. NVMe natively supports up to 64,000 queues, each with 64,000 commands, mapping 1:1 with CPU compute cores. NVMe-over-TCP maps these submission and completion queues directly into TCP socket streams:

For high-throughput clusters comparing RDMA fabric behaviors directly, review our deep-dive on NVMe-oF in Proxmox VE: SPDK vs. Kernel RoCEv2 Benchmarks.

Option 1: Configuring the Linux Kernel Target (nvmet-tcp)

The Linux kernel storage target is the recommended production baseline for high-availability Proxmox storage arrays. It is natively built into modern Linux kernels (6.8+) and provides robust resilience against network blips without crashing the user space process.

Step 1: Load Target Modules on the Storage Server

On your dedicated TrueNAS, Debian, or Proxmox storage node, load the NVMe Target kernel modules and enable persistent loading:

modprobe nvmet
modprobe nvmet-tcp
echo -e "nvmet
nvmet-tcp" >> /etc/modules-load.d/nvmet.conf

Step 2: Create Subsystem & Namespaces via nvmetcli

Install nvmetcli to manage the ConfigFS hierarchy declaratively, or generate the configuration directly via script. In this example, we export a high-performance ZFS ZVOL or raw NVMe namespace /dev/nvme0n1 over TCP port 4420:

mkdir -p /sys/kernel/config/nvmet/subsystems/nqn.2026-09.ca.cpt:nvme-san-pool1
cd /sys/kernel/config/nvmet/subsystems/nqn.2026-09.ca.cpt:nvme-san-pool1

# Configure open access (or require DH-HMAC-CHAP authentication)
echo 1 > attr_allow_any_host

# Create Namespace 1 mapped to our raw fast NVMe drive
mkdir namespaces/1
echo -n "/dev/nvme0n1" > namespaces/1/device_path
echo 1 > namespaces/1/enable

# Create TCP Listening Port on our 100GbE SAN Interface (192.168.100.10)
mkdir /sys/kernel/config/nvmet/ports/1
cd /sys/kernel/config/nvmet/ports/1
echo "ipv4" > addr_adrfam
echo "192.168.100.10" > addr_traddr
echo "tcp" > addr_trtype
echo "4420" > addr_trsvcid

# Link Subsystem to Port 1
ln -s /sys/kernel/config/nvmet/subsystems/nqn.2026-09.ca.cpt:nvme-san-pool1   /sys/kernel/config/nvmet/ports/1/subsystems/nqn.2026-09.ca.cpt:nvme-san-pool1

To ensure high availability and prevent single points of failure across your high-speed backplane, pair this configuration with our setup for Mellanox ConnectX-4 & ConnectX-5 100GbE SR-IOV in Proxmox VE.

Option 2: High-Performance SPDK Target Configuration

When maximum IOPS density and sub-10 microsecond latency are required (such as hosting high-transaction PostgreSQL databases or local AI vector stores), the Intel/Linux Foundation Storage Performance Development Kit (SPDK) is the undisputed champion. SPDK runs entirely in userspace using DPDK (Data Plane Development Kit) poll-mode drivers, completely eliminating system calls, interrupts, and context switching.

An SPDK target configuration file (nvmf.json) defines the bdevs (block devices) and TCP transport layer:

{
  "subsystems": [
    {
      "subsystem": "bdev",
      "config": [
        {
          "method": "bdev_nvme_attach_controller",
          "params": {
            "name": "Nvme0",
            "trtype": "PCIe",
            "traddr": "0000:41:00.0"
          }
        }
      ]
    },
    {
      "subsystem": "nvmf",
      "config": [
        {
          "method": "nvmf_create_transport",
          "params": {
            "trtype": "TCP",
            "max_queue_depth": 1024,
            "io_unit_size": 16384,
            "sock_priority": 6
          }
        },
        {
          "method": "nvmf_create_subsystem",
          "params": {
            "nqn": "nqn.2026-09.ca.cpt:spdk-fast-pool",
            "listen_addresses": [
              {"trtype": "TCP", "adrfam": "IPv4", "traddr": "192.168.100.10", "trsvcid": "4420"}
            ],
            "namespaces": [
              {"bdev_name": "Nvme0n1", "nsid": 1}
            ]
          }
        }
      ]
    }
  ]
}

Configuring Proxmox VE as an NVMe-oF Initiator

On your Proxmox VE compute hypervisors, the kernel NVMe fabric driver must discover and connect to the remote target. Once connected, the remote NVMe namespace appears locally in /dev/nvmeXnY just like a physically installed PCIe card.

1. Discovery and Connection via nvme-cli

# Install user-space utilities
apt-get update && apt-get install -y nvme-cli

# Discover remote target portals
nvme discover -t tcp -a 192.168.100.10 -s 4420

# Connect to target subsystem
nvme connect -t tcp -a 192.168.100.10 -s 4420 -n nqn.2026-09.ca.cpt:nvme-san-pool1

# Verify block device creation
ls -la /dev/disk/by-id/nvme-*

2. Adding the NVMe-oF Storage to Proxmox VE

To integrate the connected fabric disk into Proxmox VE’s storage pool, you can either create an LVM-thin pool over the block device for thin provisioning and snapshots, or format it as an enterprise ZFS pool. To configure LVM-thin via the Proxmox CLI:

# Create Physical Volume and Volume Group
pvcreate /dev/nvme1n1
vgcreate vg_nvme_san /dev/nvme1n1

# Create LVM-Thin Data Pool
lvcreate -l 95%FREE -T vg_nvme_san/thin_nvme_san

# Register in Proxmox /etc/pve/storage.cfg
pvesm add lvmthin NVME-SAN-STORAGE   --vgname vg_nvme_san   --thinpool thin_nvme_san   --content images,rootdir   --nodes pve-node1,pve-node2,pve-node3

For enterprise-grade arrays managing flash storage alongside hyperconverged Ceph infrastructure, compare these latency trade-offs with our analysis of Ceph RBD Block Storage vs. CephFS in Proxmox VE.

Network & Kernel Performance Tuning for 100GbE Line-Rate Storage

Running NVMe over TCP at 100GbE line rates (saturating 11,500 MB/s per link) requires specific kernel tuning to eliminate packet loss and buffer overruns:

Apply these network optimizations in /etc/sysctl.d/99-nvmeof-network.conf:

net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_congestion_control = bbr

And maximize ring buffers on your Mellanox or Intel NICs via ethtool:

ethtool -G ens1f0np0 rx 4096 tx 4096

Where to Expand Your Stack Next

• All-Flash Home Lab Fabrics: Combine NVMe-oF with enterprise data protection in our guide to NVMe-oF & ZFS dRAID in 2026: Building a 100GbE All-Flash SAN.
• 100GbE SRIOV Tuning: Maximize hypervisor throughput with Mellanox ConnectX-4 & ConnectX-5 100GbE SR-IOV in Proxmox VE.
• Software Defined Networking: Secure your storage fabrics with our blueprint for Proxmox SDN EVPN-VXLAN Isolation.

People Also Ask

Does NVMe-over-TCP require special switches or RDMA network cards?
No. NVMe-over-TCP operates over standard Ethernet networks using standard TCP/IP packets. Unlike RoCEv2 (which requires RDMA-capable NICs and switches supporting Priority Flow Control), NVMe-over-TCP runs on any commodity 10GbE, 25GbE, or 100GbE network interface card and standard managed switch.

How does NVMe-over-TCP latency compare to local NVMe SSDs?
On a well-tuned 100GbE fabric with MTU 9000, NVMe-over-TCP using the Linux kernel target adds only 18µs to 30µs of latency overhead compared to a locally attached PCIe NVMe SSD. With userspace SPDK polling targets, latency overhead drops to 9µs to 15µs.

Can I perform Proxmox VM live migration across an NVMe-oF storage pool?
Yes. Because NVMe-oF presents a shared block device visible across all nodes in the Proxmox VE cluster, Proxmox treats LVM-thin over NVMe-oF as shared cluster storage. VMs can be live-migrated between hypervisor nodes with zero disk copying and sub-second CPU handover.

Why choose NVMe-over-TCP instead of Ceph for Proxmox storage?
Ceph is a distributed, fault-tolerant object and block storage system that requires multi-node replication (typically 3x replication overhead) and substantial CPU and RAM resources. NVMe-over-TCP is a direct point-to-point block SAN protocol designed for raw, uncompromised NVMe speed and ultra-low latency without replication write amplification.