There is a specific, cold dread that hits an infrastructure team at 3:00 AM when a three-node hypervisor cluster suddenly reboots half its virtual machines without warning. You open your logging console, expecting a hardware failure or a power outage, but find neither. Instead, you see a flood of cluster membership dropouts, followed instantly by the Linux kernel hardware watchdog force-resetting your physical nodes.

Your cluster didn’t crash because of bad memory or a blown power supply. It committed suicide. Corosync lost its heartbeat for a fraction of a second, decided the nodes were partitioned, and triggered automated node fencing to prevent data corruption. The culprit? Someone started a heavy Ceph backfill operation or a virtual machine backup over the exact same network interface carrying your cluster heartbeat.

Corosync is the nervous system of Proxmox VE. It is an ultra-sensitive, real-time messaging protocol responsible for maintaining quorum and tracking cluster membership. When latency on a Corosync network spikes past a few milliseconds, the cluster panics. If you run Corosync over general-purpose VM traffic or unconstrained storage links, your cluster is a ticking time bomb.

Why Does Corosync Trigger Node Fencing on Shared Networks?

To fix cluster instability, you first have to understand the brutal economics of Proxmox quorum. Proxmox VE 8.x relies on Corosync 3.x, which uses Kronosnet (Knet) as its underlying network transport mechanism. Knet is designed to multiplex multiple logical links into a single, highly resilient network channel. It continuously sends small packet probes—heartbeats—between every node in the cluster to maintain state.

By default, Corosync’s Totem protocol expects these heartbeats to arrive reliably within very strict time windows. If a node fails to respond to heartbeat probes within the configured token timeout (typically 1000ms to 2000ms), the remaining nodes declare that node dead. To protect shared storage repositories—such as Proxmox Backup Server targets, Ceph pools, or shared SAN LUNs—from split-brain corruption, Proxmox uses a watchdog daemon (watchdog-mux or kernel-level IPMI/softdog). If a node realizes it has lost quorum, its local watchdog timer expires, and the system instantly hard-reboots the physical node.

Here is where shared networks ruin your day. If your Corosync traffic shares a physical link or VLAN with high-throughput traffic—such as Ceph East-West replication, cluster live migrations, or guest VM network spikes—the interface queue drops or delays small Corosync UDP packets. A transient latency spike over 5ms will trigger packet jitter; a spike lasting more than a second will drop cluster quorum entirely. The resulting node fencing takes down every VM running on that host.

Relying on a single network interface for Corosync, or running it across bonded interfaces that share the same physical switch infrastructure, leaves your cluster vulnerable to switch reboots, interface flapping, and micro-burst congestion.

What Are the Most Common Corosync Network Architectural Blunders?

A surprising number of enterprise deployments collapse under heavy load because administrators apply standard web-server networking concepts to cluster high-availability protocols. Corosync requires ultra-low latency, deterministic packet arrival, and fault independence above all else.

The table below breaks down how common network design patterns hold up when exposed to real-world infrastructure failures.

The major misconception here is that an LACP (802.3ad) network bond provides sufficient network redundancy for Corosync. LACP operates at Layer 2/3 and relies on hashing algorithms (such as layer2+3) to balance traffic. If the underlying switch port experiences packet loss, software hashing delays, or a spanning-tree recalculation, both links in the LACP bond can stall simultaneously. Furthermore, LACP cannot protect you against a complete top-of-rack (ToR) switch failure unless you deploy complex multi-chassis link aggregation (MLAG).

Dual Knet rings solve this problem natively at the application layer. Knet bypasses kernel bonding mechanisms by establishing independent UDP sockets across distinct physical network cards and routing them over completely isolated physical switch fabrics.

How Do Knet Link Priorities and Heartbeat Rules Actually Work?

In Corosync 3.x, Kronosnet introduces advanced transport features that allow you to define multiple interfaces (links) per node and tune how traffic routes across them. Knet supports active-active (load-balanced) and active-passive (failover) traffic policies using the knet_mode configuration option, though Proxmox VE explicitly defaults to and recommends active-passive failover via priority assignment.

When you configure multiple links in Proxmox—such as link0 (Ring 0) and link1 (Ring 1)—Knet calculates the quality and round-trip time (RTT) of each link independently. Every link is assigned a linkpriority value ranging from 0 to 255. High-priority links carry all primary heartbeat and quorum state packets. Lower-priority links remain active, continually passing lightweight ping probes to verify network viability without consuming unnecessary bandwidth.

If link0 experiences micro-packet loss or total hardware link-down events, Knet seamlessly shifts active cluster communication to link1 in sub-second time. This transition happens entirely within Kronosnet—the higher-level Corosync Totem engine never notices a failure, meaning quorum is never interrupted, no node timeouts are declared, and no fencing mechanisms are triggered.

To ensure physical failure domain isolation, you must adhere to three foundational design rules:

1. Distinct Subnets: Ring 0 and Ring 1 must reside on separate layer-3 IP subnets (e.g., 10.10.10.0/24 for Ring 0 and 10.10.20.0/24 for Ring 1). Never configure two Knet links on the same IP subnet, as kernel ARP routing will cause packet leakage across interfaces.
2. Hardware Isolation: Ring 0 should run on dedicated physical NIC ports (even 1GbE interfaces are ideal due to minimal contention), connected to Switch A. Ring 1 should run on your secondary management or cluster network, connected to Switch B.
3. Direct Connection Option: In small 2-node or 3-node clusters, Ring 0 can even be cabled directly from host to host using crossover or straight-through patch cables, completely bypassing external switch infrastructure for the primary ring.

How Do You Safely Configure Redundant Rings in /etc/pve/corosync.conf?

Modifying a live cluster’s heartbeat configuration can feel like performing open-heart surgery while running a marathon. If you make a syntax error in corosync.conf across active nodes, Corosync will fail to parse the file, lose quorum, and lock up the Proxmox Cluster File System (pmxcfs) into read-only mode.

To safely apply redundant network rings to a live Proxmox VE 8.x cluster, follow this exact implementation procedure.

Step 1: Network Interface Preparation

Verify that your secondary physical interfaces are up, configured with static IP addresses on independent subnets, and capable of communicating across hosts. Run a ping check across the secondary subnet from Node 1 to Node 2:

ping -I 10.10.20.11 10.10.20.12

Step 2: Back Up the Cluster Configuration

Proxmox manages cluster configurations through pmxcfs, which mirrors /etc/pve/corosync.conf across all nodes. Make an explicit backup before opening the configuration editor:

cp /etc/pve/corosync.conf /etc/pve/corosync.conf.bak

Step 3: Modify corosync.conf

Open /etc/pve/corosync.conf using your preferred text editor. You must perform two crucial edits: update the totem section to define multi-link parameters, and add a second link address (ring1_addr or link1) to every entry under the nodelist section.

Most importantly, you must increment the config_version value by 1. If you fail to increment this number, Corosync will ignore your changes or trigger configuration sync conflicts across the cluster.

Here is a complete production-grade configuration template for a 3-node cluster configured with dual redundant Knet rings:

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve-node-01
    nodeid: 1
    ring0_addr: 10.10.10.11
    ring1_addr: 10.10.20.11
  }
  node {
    name: pve-node-02
    nodeid: 2
    ring0_addr: 10.10.10.12
    ring1_addr: 10.10.20.12
  }
  node {
    name: pve-node-03
    nodeid: 3
    ring0_addr: 10.10.10.13
    ring1_addr: 10.10.20.13
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: Production-Cluster
  config_version: 2
  interface {
    linknumber: 0
    knet_link_priority: 150
  }
  interface {
    linknumber: 1
    knet_link_priority: 100
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

Step 4: Save and Validate Configuration Propagation

Save the file. Proxmox’s cluster filesystem will detect the file change, compare the updated config_version, and automatically broadcast the new configuration to all other nodes in the cluster within a few seconds.

Check the status of the local Corosync service to confirm that the new configuration parsed successfully without throwing syntax errors:

systemctl status corosync

What Is the 4-Step Failover Verification Protocol for Production?

Never assume redundant rings are functional simply because the configuration file parsed without errors. You must validate active link status and force real-world failover conditions during a planned maintenance window.

1. Check Real-Time Knet Link Status

Execute the low-level Corosync engine tool to inspect link metrics across all configured network channels:

corosync-cfgtool -s

The expected output must show both Link 0 and Link 1 marked as enabled and connected without transport errors:

Local node ID 1
LINK ID 0 addr 10.10.10.11 enabled connected
LINK ID 1 addr 10.10.20.11 enabled connected

2. Verify Cluster Quorum State

Query the high-level Proxmox cluster manager to confirm quorum integrity and active node votes:

pvecm status

Ensure that the output confirms the cluster is fully quorate, listing total votes, expected votes, and all active node members.

3. Simulate Interface Failure (Non-Destructive Soft Test)

To verify that Kronosnet seamlessly handles instantaneous link loss, administrative link down commands can be used on the primary interface (Ring 0) on a single node while monitoring the cluster log stream on another node:

# On Node 2 (Monitor logs in real-time):
journalctl -u corosync -f

On Node 1 (Disable primary Ring 0 interface):
ip link set dev eth0 down

Inspect the output on Node 2. You will observe Kronosnet flagging Link 0 on Node 1 as down, followed immediately by traffic seamlessly maintaining quorum over Link 1. Crucially, no node reboots, cluster quorums drops, or VM interruptions should occur.

4. Restore Link and Verify Automatic Recovery

Re-enable the primary interface on Node 1:

ip link set dev eth0 up

Re-run corosync-cfgtool -s. Kronosnet should automatically re-establish the connection on Link 0 and restore it as the primary active pathway due to its higher knet_link_priority setting (150 vs 100).

How Do You Handle Even-Node Clusters with an External QDevice?

Configuring redundant network rings protects your cluster against physical cabling and switch failures, but it cannot solve mathematical split-brain dilemmas caused by even-node counts. In a 2-node or 4-node cluster, if physical network separation splits the cluster perfectly in half (e.g., 1 node vs 1 node), neither side can achieve a strict majority (>50% of total votes). Both sides lose quorum and immediately fence themselves, leaving your entire infrastructure offline.

To resolve this, deploy an external Corosync Quartermaster Device (QDevice). A QDevice is a lightweight daemon (corosync-qnetd) hosted on an external Linux machine, Raspberry Pi, or cloud virtual machine residing outside the main hypervisor cluster.

The QDevice acts as an independent tie-breaker vote. In a 2-node cluster, the total vote count becomes 3 (Node 1 + Node 2 + QDevice = 3 votes). If Node 1 loses communication with Node 2, but can still reach the external QDevice over the management network, Node 1 retains 2 out of 3 votes (a clear majority). Node 1 stays online and keeps VMs running, while Node 2 safely fences itself.

Setting up a QDevice in Proxmox VE 8.x requires just two simple steps:

1. Install the daemon on an external Debian/Ubuntu machine:

   apt-get install corosync-qnetd

2. Initialize and join the QDevice from any active Proxmox node:

   pvecm qdevice setup <qnetd-ip-address>

Once joined, run pvecm status to verify that the Qdevice line appears in your cluster summary, showing active voting membership. Combined with dual redundant Knet rings, your Proxmox cluster gains true enterprise-grade resilience against both network disruptions and split-brain scenarios.

Frequently Asked Questions