You’ve just invested in a powerful new dual-socket server, perhaps an AMD EPYC or an Intel Xeon, loaded with DDR5 memory, expecting it to be a performance beast for your high-throughput databases or latency-sensitive applications. You fire it up, deploy your workload, and then… you see it. Inconsistent latency spikes, throughput that doesn’t scale as expected, or perhaps even inexplicable CPU utilization issues. It feels like you’re leaving a significant chunk of performance on the table, despite pouring capital into top-tier hardware. You’re not alone in this frustration; it’s a common, costly trap that many fall into, often without realizing the underlying culprit: Non-Uniform Memory Access, or NUMA.
The promise of multi-socket systems is immense, but without understanding how memory is physically organized and accessed, you’re essentially running a finely tuned race car with a flat tire. The good news is, this isn’t some esoteric academic problem; it’s a fundamental architectural reality that, once understood, can be tamed to unlock the full potential of your hardware. We’re going to demystify NUMA, expose the common pitfalls, and equip you with the practical knowledge to optimize your Linux systems for predictable, high-performance memory access.
For high-performance, latency-sensitive Linux workloads on multi-socket servers, disable automatic kernel NUMA balancing. Instead, manually pin processes and their memory to specific NUMA nodes using tools like numactl and taskset to ensure local memory access, drastically reducing latency and improving throughput predictability.
Why Your Multi-Socket Server Isn’t Just One Big Pool of Memory
Let’s start with the fundamental architecture. When you look at a modern dual-socket server, you’re not looking at two CPUs sharing a single, monolithic block of RAM. That’s the costly trap many fall into. Instead, each processor socket (CPU) on the motherboard has its own dedicated set of DDR5 memory channels. This memory is local to that specific socket. Think of it like two separate, high-speed memory banks, each directly connected to its own CPU.
These individual CPU sockets are then interconnected by incredibly fast, coherent links: AMD’s Infinity Fabric or Intel’s Ultra Path Interconnect (UPI). These interconnects allow a core on CPU0 to access memory that’s physically attached to CPU1, and vice-versa. This is where the “Non-Uniform” part of NUMA comes in.
Accessing memory that’s local to the CPU core currently executing a process is incredibly fast. It’s like reaching for a book on your desk. But when that same core needs to access memory attached to the other socket, it has to traverse the Infinity Fabric or UPI interconnect. This “remote” memory access introduces a significant latency penalty. We’re talking about 30% to 50% higher memory latency compared to local access. Not only that, but frequent remote access can saturate the interconnect bandwidth, creating a bottleneck that affects all processes trying to communicate across sockets.
Imagine trying to run a PostgreSQL database where every other memory access has to travel across the server, adding tens or even hundreds of nanoseconds to each operation. For a database that thrives on predictable, low-latency memory access, this can be catastrophic, leading to query slowdowns, transaction delays, and overall inconsistent performance. Your expensive hardware is effectively being hobbled by a lack of architectural awareness.
The Double-Edged Sword of Automatic NUMA Balancing: Why the Kernel Can Hurt Your Performance
Linux, in its wisdom, attempts to mitigate NUMA issues automatically. The kernel.numa_balancing feature, enabled by default on many modern distributions, is designed to dynamically migrate pages of memory to the NUMA node closest to the CPU accessing them. On the surface, this sounds fantastic – the kernel handles the complexity, and your applications just run faster, right?
Unfortunately, for high-performance, latency-sensitive workloads, this automatic balancing often does more harm than good. Here’s why:
- CPU Overhead: The kernel doesn’t have a crystal ball. To decide which memory pages to migrate, it needs to constantly monitor memory access patterns. This monitoring isn’t free; it involves background page scanning and tracking, which consumes CPU cycles. For applications that are already CPU-bound or require every cycle for their primary task, this overhead is a direct performance hit.
- Unpredictable Latency Jitter: Memory migration isn’t instantaneous. When the kernel decides to move a page, it involves copying data, updating page tables, and potentially pausing the accessing process briefly. This introduces unpredictable latency spikes, or “jitter,” into your application’s execution. For databases like PostgreSQL, Redis, or MySQL, which rely on consistent, low-latency responses, this jitter can be devastating. A query that normally takes 5ms might suddenly take 50ms because a critical page was being migrated.
- Thrashing: In some scenarios, especially with workloads that have complex or rapidly changing memory access patterns, the kernel might get into a state where it’s constantly migrating pages back and forth between nodes, trying to optimize, but instead just wasting CPU cycles and interconnect bandwidth. This “NUMA thrashing” can lead to worse performance than if no balancing were attempted at all.
For these reasons, the vast majority of high-performance database administrators and systems engineers disable kernel.numa_balancing and opt for a manual, explicit approach to NUMA management. They understand that while the kernel’s intentions are good, its general-purpose approach isn’t suitable for specialized, demanding workloads.
Pinning Down Performance: Mastering numactl and taskset for Predictable Latency
If automatic balancing is a trap, then manual pinning is your escape route. The goal is simple: ensure that a process’s threads run on CPU cores within a specific NUMA node, and that the memory those threads use is allocated from the local memory of that same NUMA node. This eliminates remote memory access penalties and interconnect saturation.
The primary tools for this are numactl and taskset.
numactl: This utility allows you to run a command with a specific NUMA policy. It’s incredibly powerful because it can control both CPU affinity (which cores a process runs on) and memory affinity (which NUMA node’s memory a process allocates from).--cpunodebind=nodes: This binds the process to run only on CPUs within the specified NUMA nodes. For example,--cpunodebind=0would restrict the process to cores on NUMA node 0.--membind=nodes: This binds the process to allocate memory only from the specified NUMA nodes. If the process tries to allocate memory and the specified nodes are full, the allocation will fail. This is a strict policy. For example,--membind=0would ensure all memory comes from node 0.--interleave=nodes: This is a more flexible memory policy. It interleaves memory allocations across the specified NUMA nodes. This can be useful for workloads that are truly distributed across multiple nodes and benefit from spreading memory access, but it still introduces remote access for some pages. For most latency-sensitive applications,--membindis preferred.--localalloc: This policy attempts to allocate memory from the node where the process is currently running. This is often a good default if you’re only binding CPU affinity and want the kernel to do its best to keep memory local.
Example: To run a PostgreSQL instance on NUMA node 0, using only memory from NUMA node 0:
numactl --cpunodebind=0 --membind=0 postgres -D /var/lib/pgsql/data
taskset: Whilenumactlcan set CPU affinity,tasksetis a more direct and often simpler tool for just CPU affinity. It allows you to set or retrieve the CPU affinity mask of a process. This is useful when you want to restrict a process to a specific set of CPU cores, regardless of their NUMA node, or as a complementary tool tonumactlfor fine-grained CPU control.
Example: To run a CPU-intensive calculation on CPUs 0-3:
taskset -c 0-3 my_calculation_program
When combining with numactl, you might use taskset to bind to specific cores within a NUMA node, and numactl to ensure memory is local to that node. For instance, if NUMA node 0 has cores 0-15:
numactl --membind=0 taskset -c 0-7 my_app
The key is to understand your application’s resource requirements. Does it fit entirely within a single NUMA node’s CPU and memory capacity? If so, strict binding to one node is ideal. If it’s a very large application that must span nodes, then careful consideration of --interleave or splitting the application into multiple, separately pinned instances might be necessary.
The Hidden Cost of Unmanaged Hugepages: Preventing Cross-Socket Memory Fragmentation
Hugepages (large memory pages, typically 2MB or 1GB) are a critical optimization for many high-performance applications, especially databases and virtualization platforms. They reduce Translation Lookaside Buffer (TLB) misses, leading to faster memory access. However, if not managed correctly in a NUMA environment, they can introduce their own set of problems.
By default, when you configure vm.nr_hugepages, the kernel allocates these hugepages across all available NUMA nodes in an interleaved fashion. While this might seem balanced, it can lead to cross-socket memory fragmentation. If your application is pinned to NUMA node 0, but some of its required hugepages were allocated on NUMA node 1, you’re back to square one with remote memory access penalties.
The solution is to explicitly allocate hugepages per NUMA node. This is done using the vm.nr_hugepages_per_node sysctl parameter.
Example: To allocate 1024 hugepages (2MB each) specifically on NUMA node 0, and another 1024 on NUMA node 1:
echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
Or, for a more persistent configuration via sysctl.conf or a .conf file in /etc/sysctl.d/:
vm.nr_hugepages_per_node=node0:1024,node1:1024
This ensures that when your application, pinned to NUMA node 0, requests hugepages, it gets them from the local pool, maintaining optimal performance.
Comparing NUMA Strategies: Auto-Balancing vs. Manual Pinning
Let’s put it into perspective. When do you use what?
| Feature/Strategy | Automatic NUMA Balancing (kernel.numa_balancing=1) |
Manual NUMA Pinning (kernel.numa_balancing=0) |
Best Use Case | Key Considerations |
|---|---|---|---|---|
| Complexity | Low (kernel handles it) | High (requires explicit configuration) | General purpose servers, desktop environments | Performance-critical applications, databases, HPC |
| Performance Predictability | Low (latency jitter due to migrations) | High (consistent local memory access) | Workloads tolerant to occasional latency spikes | Workloads requiring strict QoS and low latency |
| CPU Overhead | Present (background scanning, migration) | Minimal (no active kernel balancing) | Applications with ample CPU headroom | CPU-bound applications where every cycle counts |
| Memory Allocation | Dynamic, kernel-managed migration | Static, user-defined (e.g., --membind) |
Applications with unpredictable memory access patterns | Applications with stable, predictable memory footprints |
| Hugepage Management | Allocated interleaved across nodes by default | Explicitly allocated per node (vm.nr_hugepages_per_node) |
Not applicable for hugepage-intensive workloads | Essential for hugepage-enabled, performance-critical apps |
Your 4:00 PM Playbook: A Step-by-Step Guide to NUMA Optimization
Ready to take control? Here’s your actionable checklist to optimize your Linux server for NUMA performance. This isn’t just theory; it’s what the pros do.
Step 1: Understand Your Server’s NUMA Topology
Before you change anything, you need to know what you’re working with.
- Identify NUMA nodes, CPUs, and memory:
numactl --hardware
This command will show you the number of NUMA nodes, the CPUs associated with each node, and the amount of memory available on each node. Pay close attention to the “distance” between nodes. - Get detailed CPU information:
lscpu
Look for “NUMA node(s)” and “NUMA nodeX CPU(s)” to confirm which logical CPUs belong to which physical NUMA node.
Step 2: Disable Automatic NUMA Balancing
For high-performance workloads, this is almost always the first step.
- Temporarily disable:
echo 0 > /proc/sys/kernel/numa_balancing - Persistently disable (recommended):
Edit/etc/sysctl.confor create a new file like/etc/sysctl.d/99-numa.confand add:
kernel.numa_balancing = 0
Then apply the changes:sysctl -p - Verify:
cat /proc/sys/kernel/numa_balancing(should output0)
Step 3: Pin Your Workloads to Specific NUMA Nodes
This is where you tell your applications exactly where to run and where to get their memory.
- For individual processes (e.g., a database instance):
Usenumactlwhen starting the application. For example, to bind a PostgreSQL instance to NUMA node 0:
numactl --cpunodebind=0 --membind=0 /usr/bin/pg_ctl -D /var/lib/pgsql/data start - For systemd services:
Edit the systemd unit file (e.g.,/etc/systemd/system/your-app.service). AddCPUAffinityandNUMAPolicydirectives to the[Service]section:
“`
[Service]
ExecStart=/usr/bin/your-app-binary
CPUAffinity=0-15 # Cores belonging to NUMA node 0
NUMAPolicy=bind
NUMAMask=0 # Bind to NUMA node 0
“`
Then reload systemd and restart your service:
systemctl daemon-reload
systemctl restart your-app.service - For multi-threaded applications spanning nodes (rare, use with caution):
Consider--interleave=0,1if your application genuinely benefits from spreading memory across nodes, but understand the latency implications. For most, strict binding is better.
Step 4: Configure Hugepages for NUMA (If Applicable)
If your application uses hugepages (e.g., Oracle, Redis, some JVMs), manage them per-node.
- Determine required hugepage size and quantity.
- Persistently configure per-node hugepages:
Edit/etc/sysctl.confor create a new file in/etc/sysctl.d/.
For 2MB hugepages:
vm.nr_hugepages_per_node=node0:1024,node1:1024(allocates 1024 hugepages on node 0, 1024 on node 1)
Then apply:sysctl -p - Verify hugepage allocation:
cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
And check overall:grep HugePages_Total /proc/meminfo
Step 5: Monitor and Validate Your Configuration
Optimization is an iterative process. Don’t just set it and forget it.
- Check NUMA statistics:
numastat -m(shows memory usage per NUMA node)
numastat -c(shows NUMA stats for a specific process)
Look fornuma_hit(local access) andnuma_foreignornuma_miss(remote access). Ideally,numa_foreignandnuma_missshould be very low or zero for pinned processes. - Use
perffor deeper insights:
perf stat -e 'mem_load_retired.l3_miss_remote_dram' -p
This can help quantify remote memory access. - Application-level metrics:
Most importantly, monitor your application’s actual performance metrics: query latency, transaction throughput, response times. These are the ultimate indicators of success.
Step 6: Document and Automate
Once you’ve found an optimal configuration, document it thoroughly. Use configuration management tools (Ansible, Puppet, Chef) to automate the deployment of your sysctl settings and systemd unit file modifications across your fleet. This ensures consistency and reproducibility.
By following this playbook, you’re not just tweaking parameters; you’re fundamentally aligning your software’s memory access patterns with your hardware’s physical architecture. This deep understanding and deliberate configuration are what separate good performance from truly exceptional, predictable performance on multi-socket Linux systems. It’s about taking control, understanding the nuances, and making your expensive hardware work exactly as it was designed to.
Frequently Asked Questions
Why is kernel.numa_balancing enabled by default if it’s often detrimental?
kernel.numa_balancing is a general-purpose feature designed to improve performance for a wide range of common workloads, especially those that aren’t explicitly NUMA-aware or don’t have strict latency requirements. For typical desktop use or less demanding server applications, it can provide a net benefit by automatically migrating pages to where they’re most frequently accessed. However, for highly optimized, latency-sensitive applications like databases or HPC, its overhead and unpredictable jitter outweigh these benefits.
Can I use numactl with Docker or Kubernetes?
Yes, you absolutely can. For Docker, you can pass numactl as part of your container’s entrypoint command. For Kubernetes, you’d typically use a Pod‘s resources.limits.cpu and resources.limits.memory combined with a topologySpreadConstraints or a nodeSelector to ensure the pod lands on a specific NUMA-aware node. More advanced NUMA pinning within containers often requires a kubelet feature called Topology Manager or custom operators that leverage numactl within the container’s command or args.
What happens if I --membind to a NUMA node that runs out of memory?
If a process is strictly bound with --membind=nodes and attempts to allocate memory when the specified NUMA node(s) are full, the memory allocation will fail. This can lead to application crashes or out-of-memory errors, even if other NUMA nodes on the server have free memory. This is why careful planning of memory capacity per node and application requirements is crucial when using strict memory binding.
How do I know which CPUs belong to which NUMA node?
You can use numactl --hardware which provides a summary of NUMA nodes and their associated CPUs. For a more detailed view, lscpu will list each logical CPU and its corresponding NUMA node. For example, lscpu | grep "NUMA node" will show you lines like “NUMA node0 CPU(s): 0-15” and “NUMA node1 CPU(s): 16-31”, clearly mapping CPU ranges to nodes.

