You’ve poured your heart and soul into building a robust Proxmox homelab or production cluster. You’ve optimized storage, tuned your CPU governors, and meticulously crafted your VM configurations. Yet, when it comes to network performance, you still feel that nagging whisper: “Is this really as fast as it could be?” You spin up a high-throughput application, a demanding firewall, or a critical database, and despite all your efforts, the network just… isn’t quite bare metal. It’s good, sure. VirtIO is fantastic for general-purpose workloads, but you know there’s a ceiling, a subtle latency penalty, a CPU overhead that shouldn’t be there when you’re pushing serious packets. You’re investing in 25GbE or even 100GbE Mellanox ConnectX-5 adapters, but are you truly unlocking their full potential within your virtualized environment? The frustration is real when your expensive network hardware feels bottlenecked by software abstraction. You want direct, uncompromised speed, and you want it now.

Why Your Virtual Network Isn’t Performing Like Bare Metal (And What Most Get Wrong)

Let’s be frank: most Proxmox users, even experienced ones, settle for “good enough” networking. They configure a standard Linux bridge (`vmbr0`), attach VirtIO network devices to their VMs, and call it a day. For 90% of workloads – web servers, file shares, even light database instances – this setup is perfectly adequate. VirtIO is efficient, well-supported, and offers excellent performance for its class. But when you start pushing the boundaries, when you need every single packet delivered with minimal latency and maximum throughput, when you’re dealing with 25GbE or 100GbE adapters, that “good enough” quickly becomes a costly trap.

Here’s what most practitioners get wrong: they assume that because their physical NIC is fast, their virtual NICs will automatically inherit that speed without any further optimization. They overlook the fundamental overheads inherent in traditional virtualized networking. Every packet sent or received by a VirtIO device in a VM has to traverse the hypervisor’s network stack. This involves context switching between the VM and the host kernel, multiple memory copies, and significant CPU cycles on the host to manage the virtual network bridge. It’s a brilliant abstraction, but it’s still an abstraction, and abstractions come with a cost.

Imagine you’re trying to deliver a package. With VirtIO, you hand the package to a hypervisor agent, who then takes it to the physical post office (your NIC). With SR-IOV, you hand the package directly to a dedicated postal worker (a Virtual Function) who has direct access to the physical post office. The difference in efficiency is profound.

This is where Single Root I/O Virtualization (SR-IOV) steps in, fundamentally changing the game. SR-IOV allows a single physical PCI Express (PCIe) device, like your Mellanox ConnectX-5, to present itself as multiple separate, lightweight PCIe devices to the hypervisor. These are called Virtual Functions (VFs). Each VF is essentially a slice of the physical hardware, complete with its own memory-mapped registers and interrupts. When you assign a VF directly to a VM, that VM gains direct, unmediated access to a portion of the physical network adapter.

The magic happens because the data path completely bypasses the host’s network stack. The VM communicates directly with the hardware VF via IOMMU (Input/Output Memory Management Unit), which handles memory translations and isolation. This means:

* Bare-metal performance: Near line-rate throughput and sub-microsecond latency, indistinguishable from a physical machine.
* Zero host CPU overhead: The host CPU is no longer involved in packet processing for that VM’s network traffic. This frees up valuable CPU cycles for other tasks or other VMs.
* Reduced latency and jitter: No more context switching or software bridge processing.

Your Mellanox ConnectX-5 EN adapters are perfectly suited for SR-IOV. These cards are designed for high-performance computing, low-latency networking, and virtualization acceleration. They can expose a significant number of VFs (up to 127 per port, though practical limits often depend on firmware and system resources), making them ideal for dense virtualization environments where network performance is paramount.

Let’s put this into perspective with a comparison:

The trade-off for this incredible performance is complexity and the loss of live migration for VMs using SR-IOV VFs. Since the VF is directly tied to physical hardware, it cannot simply be moved to another host without downtime. For critical network appliances or high-performance workloads where downtime is acceptable for maintenance, or where the VM is designed to be highly available through other means (e.g., CARP for firewalls), SR-IOV is an unparalleled solution.

Before we dive into the actionable steps, ensure your Proxmox host meets the fundamental prerequisites:
1. IOMMU Support: Your CPU (Intel VT-d or AMD-Vi) and motherboard BIOS must support and have IOMMU enabled. This is non-negotiable for SR-IOV.
2. Compatible NIC: Your Mellanox ConnectX-5 EN adapter (or similar) must support SR-IOV. Most modern enterprise-grade NICs do.
3. Proxmox VE 8.x: While SR-IOV works on older versions, we’re focusing on the latest stable release for best compatibility and features.

Getting Your Proxmox Host Ready: Enabling IOMMU for Direct Hardware Access

The first crucial step is to ensure your Proxmox host can actually see and manage direct hardware access for your VMs. This is handled by the IOMMU. Without it, SR-IOV simply won’t work.

1. Verifying and Enabling IOMMU Support:

Start by checking your BIOS/UEFI settings. Look for options like “Intel VT-d,” “AMD-Vi,” “Virtualization Technology for Directed I/O,” or similar. Ensure these are enabled. The exact wording varies by motherboard manufacturer. Save and exit.

Next, we need to tell the Linux kernel on your Proxmox host to activate IOMMU. This is done via GRUB parameters.

* Edit GRUB configuration:
Open the GRUB default file:
“`bash
nano /etc/default/grub
“`
* Modify `GRUB_CMDLINE_LINUX_DEFAULT`:
Locate the line starting with `GRUB_CMDLINE_LINUX_DEFAULT`.
For Intel CPUs, add `intel_iommu=on iommu=pt` inside the quotes.
For AMD CPUs, add `amd_iommu=on iommu=pt` inside the quotes.
The `iommu=pt` (pass-through) parameter is crucial as it optimizes IOMMU for direct device assignment, which is what SR-IOV leverages.

Example for Intel:
“`
GRUB_CMDLINE_LINUX_DEFAULT=”quiet intel_iommu=on iommu=pt”
“`
Example for AMD:
“`
GRUB_CMDLINE_LINUX_DEFAULT=”quiet amd_iommu=on iommu=pt”
“`
* Update GRUB and reboot:
After saving the file, update GRUB and reboot your Proxmox host:
“`bash
update-grub
reboot
“`
* Verify IOMMU is active:
Once the host reboots, log in and verify that IOMMU is enabled and working correctly:
“`bash
dmesg | grep -e DMAR -e IOMMU
“`
You should see output indicating IOMMU is enabled, like `DMAR: IOMMU enabled`.

Also, check if IOMMU groups are being created:
“`bash
find /sys/kernel/iommu_groups/ -type l
“`
This command should list various IOMMU groups, confirming that your devices are correctly grouped for pass-through. Your Mellanox NIC should ideally be in its own IOMMU group, or at least a group with devices you intend to pass through together.

Configuring Your Mellanox ConnectX-5: Exposing Virtual Functions

With IOMMU enabled, the next step is to configure your Mellanox ConnectX-5 adapter to actually expose the Virtual Functions. This is a firmware-level setting.

2. Updating Firmware and Enabling VFs with `mstflint`:

Mellanox provides a powerful set of tools called MLNX_OFED (Mellanox OpenFabrics Enterprise Distribution) which includes `mstflint` for firmware management. While installing the full MLNX_OFED stack is an option, for just firmware updates and SR-IOV configuration, `mstflint` is often sufficient and less intrusive on a Proxmox host.

* Install `mstflint`:
You might need to install `mstflint` if it’s not already present. It’s usually part of the `mellanox-fw-updater` package or can be extracted from the MLNX_OFED ISO.
“`bash
apt update
apt install mstflint # Or download from Mellanox website if not in repos
“`
If `mstflint` isn’t readily available via `apt`, you’ll need to download the MLNX_OFED driver package from the NVIDIA/Mellanox website, extract it, and find the `mstflint` utility within. A common path is `MLNX_OFED_LINUX-––/DEBS/mlnx-fw-updater__amd64.deb`. You can install this specific debian package.

* Identify your Mellanox device:
First, find the PCI address of your ConnectX-5 card.
“`bash
lspci -nn | grep Mellanox
“`
You’ll see output like:
`81:00.0 Ethernet controller [0200]: Mellanox Technologies ConnectX-5 Ex EN`
`81:00.1 Ethernet controller [0200]: Mellanox Technologies ConnectX-5 Ex EN`
Note down the PCI addresses (e.g., `81:00.0`, `81:00.1`).

* Query current firmware settings:
Use `mstflint` to query the current SR-IOV settings for your device. Replace `0000:81:00.0` with your actual PCI address.
“`bash
mstflint -d 0000:81:00.0 q
“`
Look for lines related to `SRIOV_EN` and `NUM_OF_VFS`. `SRIOV_EN` should be `True`, and `NUM_OF_VFS` will show the current number of VFs.

* Enable SR-IOV and set number of VFs:
Now, we’ll enable SR-IOV and specify how many Virtual Functions you want to expose per port. A common starting point is 8 VFs per port, but you can go higher depending on your needs and system resources (up to 127 per port on ConnectX-5).
Important: You need to do this for each port of your dual-port card.

“`bash
mstflint -d 0000:81:00.0 set_sriov_config NUM_OF_VFS=8 SRIOV_EN=true
mstflint -d 0000:81:00.1 set_sriov_config NUM_OF_VFS=8 SRIOV_EN=true
“`
Double-check the PCI addresses! If you only have one port, only run the command for that port.

* Burn the new firmware:
After setting the configuration, you need to burn these changes to the NIC’s firmware.
“`bash
mstflint -d 0000:81:00.0 -i /dev/null -y burn
mstflint -d 0000:81:00.1 -i /dev/null -y burn
“`
The `-i /dev/null` is a workaround for `mstflint` requiring an image file, even when just burning configuration changes. The `-y` confirms the action without prompting.

* Reboot the host:
A full system reboot is required for the firmware changes to take effect.
“`bash
reboot
“`
* Verify VF creation:
After reboot, verify that the VFs have been created.
“`bash
lspci -nn | grep Mellanox
“`
You should now see your original physical functions (PFs) and multiple new entries for Virtual Functions. For example, if you set `NUM_OF_VFS=8` for `81:00.0`, you’d see `81:00.0` (PF) and `81:00.2` through `81:00.9` (VFs). The exact numbering can vary slightly.

You can also check the network interfaces:
“`bash
ip link show
“`
You’ll see interfaces like `enp129s0f0` (PF) and then `enp129s0f0np0` (VF0), `enp129s0f0np1` (VF1), etc., indicating the VFs are present.

Automating SR-IOV VF Creation on Proxmox Boot

The VFs created by the firmware are ephemeral in the sense that the Linux kernel needs to instantiate them. While the firmware enables the capability, the OS needs to tell the driver to create them. Often, after a reboot, the VFs might not be automatically instantiated by the kernel. We need a reliable way to ensure they are always available.

3. Persistent VF Creation with Systemd:

We’ll create a simple script and a systemd service to ensure VFs are instantiated every time your Proxmox host boots.

* Create a script to instantiate VFs:
Create a new script file:
“`bash
nano /usr/local/bin/create_vfs.sh
“`
Add the following content. Replace `enp129s0f0` and `enp129s0f1` with the actual names of your Mellanox Physical Function (PF) interfaces. You can find these using `ip link show`.
“`bash
#!/bin/bash

# Define your Mellanox Physical Function (PF) interfaces
# Use `ip link show` to find the correct names (e.g., enp129s0f0, enp129s0f1)
PF_INTERFACES=(“enp129s0f0” “enp129s0f1”) # Adjust as needed for your card’s ports

# Number of VFs to create per PF (must match firmware setting)
NUM_VFS=8

echo “Ensuring SR-IOV VFs are created for Mellanox ConnectX-5…”

for PF_IFACE in “${PF_INTERFACES[@]}”; do
if ip link show “$PF_IFACE” &> /dev/null; then
echo “Processing PF: $PF_IFACE”

# Disable SR-IOV first to ensure a clean slate
echo 0 > “/sys/class/net/$PF_IFACE/device/sriov_numvfs”
sleep 1 # Give it a moment

# Enable SR-IOV and create VFs
echo “$NUM_VFS” > “/sys/class/net/$PF_IFACE/device/sriov_numvfs”
echo “Created $NUM_VFS VFs for $PF_IFACE”
else
echo “Warning: PF interface $PF_IFACE not found. Skipping.”
fi
done

echo “SR-IOV VF creation process complete.”
“`
Make the script executable:
“`bash
chmod +x /usr/local/bin/create_vfs.sh
“`

* Create a systemd service file:
Create the service definition:
“`bash
nano /etc/systemd/system/mlx-sriov.service
“`
Add this content:
“`
[Unit]
Description=Mellanox ConnectX-5 SR-IOV VF Creation
After=network.target pve-cluster.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/create_vfs.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
“`
The `After=network.target pve-cluster.service` ensures the network is up and Proxmox services are initialized before we try to configure the VFs. `RemainAfterExit=yes` is useful for `oneshot` services that perform a setup task.

* Enable and start the service:
Reload systemd, enable the service to run on boot, and then start it immediately:
“`bash
systemctl daemon-reload
systemctl enable mlx-sriov.service
systemctl start mlx-sriov.service
“`
* Verify service status and VFs:
Check if the service ran successfully:
“`bash
systemctl status mlx-sriov.service
“`
And confirm the VFs are present again:
“`bash
lspci -nn | grep Mellanox | grep Virtual
“`
You should see your VFs listed.

Assigning SR-IOV Virtual Functions to Your VMs

Now for the payoff! With IOMMU enabled, firmware configured, and VFs persistently created, you can finally assign these direct hardware slices to your virtual machines.

4. Attaching VFs to Virtual Machines:

You can attach VFs using either the Proxmox web GUI or the command-line interface.

* Via Proxmox Web GUI:
1. Navigate to your VM in the Proxmox web interface.
2. Go to the “Hardware” tab.
3. Click “Add” -> “PCI Device”.
4. In the “PCI Device” dialog, select your Mellanox ConnectX-5 Virtual Function from the “Raw Device” dropdown list. You’ll see entries like `Host PCI device 0000:81:00.2 (Mellanox Technologies ConnectX-5 Ex EN Virtual Function)`.
5. Crucially, check the “Primary GPU” checkbox if you are passing through a GPU, but for network cards, ensure “All Functions” is unchecked (you’re passing a specific VF, not the whole card) and “PCI-Express” is checked for better performance.
6. Click “Add”.

* Via Proxmox CLI:
You can also add the PCI device using the `qm set` command. First, identify the PCI address of the specific VF you want to assign.
“`bash
lspci -nn | grep Mellanox | grep Virtual
“`
Let’s say you want to assign `0000:81:00.2` to VMID `101`.
“`bash
qm set 101 -hostpci0 0000:81:00.2,pcie=1
“`
The `pcie=1` option tells Proxmox to use a PCI-Express slot for the device, which is generally recommended for modern hardware and better performance. You can increment `hostpci0` to `hostpci1`, `hostpci2`, etc., if you’re adding multiple PCI devices.

* Guest OS Considerations:
Once the VF is assigned, start your VM. Inside the guest operating system, you will see a