mailwal

CXL Memory Bandwidth: Weighted Interleave Explained

Engineering Deep Dives

Welcome to Engineering Deep Dives — deep, practical content for engineers who want to go beyond the surface.

CXL Memory Bandwidth: Weighted Interleave Explained

CXL is sold as more gigabytes, but its better use is often more bandwidth. How weighted interleave works, when hot pages on CXL win, and when they hurt.

Capacity Is the Boring Half

Two workloads run on the same Sapphire Rapids server, with local DDR5 and a single CXL memory expander. Sun and colleagues measured both in their Demystifying CXL study, moving a growing share of each workload’s pages onto the CXL node.

The first is the embedding-reduction stage of DLRM, a recommendation model, whose throughput is bounded by memory bandwidth. With every page in local DDR it stops scaling past about 20 threads. At 32 threads, putting 63% of its pages on the slower CXL memory gave 88% more throughput than keeping them all local. The second is Redis, a latency-bound key-value store, and for Redis every step in the same direction cost throughput: 8% with a quarter of its pages on CXL, 30% with all of them.

Same machine, same knob, opposite answers. We will spend the rest of this part working out why. If you read the previous article as a story about keeping hot pages close, the first result looks like a measurement error. It is not one. It is the other half of what CXL memory is for.

The core insight: capacity tiering asks which cold pages can afford to live far away. Bandwidth tiering asks the opposite question, how many hot pages to send to the slow node so that both memory systems stay busy. It pays only when a workload is bound by bandwidth, it needs ratios rather than rankings, and on a latency-bound workload the very same move is pure loss.

Two models of the same memory

Most of what gets written about CXL uses the capacity model, and so did most of this series so far. In it, local DRAM is the good memory and CXL is the overflow. The job of policy is to keep the hot set local and push cold data outward, which is exactly the classifier the previous article dissected. Success means as few accesses as possible reach the CXL node.

The bandwidth model starts somewhere else. It treats the DDR channels and the CXL link as two pipes into the same CPU, each with its own ceiling. If a workload can saturate the first pipe, the second one is free bandwidth, and leaving it idle wastes it. Success means both pipes run close to their limits, which necessarily sends a meaningful share of busy traffic to the slower node.

The same hardware, two opposite goals. The capacity model ranks pages; the bandwidth model splits traffic.

The first article in this series made the economic case in pins. A DDR channel needs hundreds of pins, whereas a CXL link of eight lanes roughly matches one DDR5-4800 channel on a fraction of them, and Kadiyala and Daglis show per-core memory bandwidth shrinking on many-core parts. CXL for People Who Write Software, Not Silicon mentioned that this makes CXL a bandwidth expander as well as a capacity one. The question for us is what software must do to collect on that.

It helps to see how much is at stake in usable rather than peak terms. Suppose your socket has eight DDR5-4800 channels, about 307 GB/s at peak. If you respect the 60% comfort line that Meta’s study reports for production services, you can use about 184 GB/s before latency starts to climb. One CXL device worth roughly one more channel adds about 38 GB/s of peak, or about 23 GB/s at the same comfort line, so usable bandwidth grows by roughly 12%. That is not a revolution. For a service pinned against its bandwidth ceiling, though, 12% more headroom may be the difference between adding a server and not, and the gain grows as more CXL devices share the socket.

Why moving hot pages further away can make them faster

The intuition that farther memory is always slower comes from measuring one access at a time on an idle machine. Under load the picture changes, because the time for an access includes waiting in line at the memory controller.

Sun and colleagues describe this directly while building their placement tuner. DLRM’s throughput tracked the memory bandwidth it consumed, until consumption passed a certain point. Beyond it, memory access latency rose rapidly because of contention and queueing at the memory controller, and throughput fell with it. Meta’s memory-subsystem study reports the same shape from production: services rarely run their memory above about 60% bandwidth utilisation, since pushing further makes latency rise steeply, so the authors count anything above 60% as bandwidth-bound and anything between 40% and 60% as bandwidth-sensitive, prone to transient peaks and tail spikes.

That curve is why the counter-intuitive move works. Shifting some traffic to the CXL node shortens the queue at the DDR controllers, so the remaining DDR accesses get faster. Each access that goes to CXL is individually slower, yet the average can fall, as long as the CXL side has not yet built a queue of its own. The CXL introduction by Das Sharma, Blankenship and Berger states the same idea in one line: considering loaded rather than idle latency, CXL may perform better than DDR thanks to its bandwidth.

There is a simple identity underneath all of this that you can check with a calculator. The bandwidth a thread draws equals the bytes it has in flight divided by how long each request takes. Suppose a thread keeps ten 64-byte cache misses outstanding and each takes 100 ns; it moves about 6.4 GB/s. To draw 300 GB/s you would need roughly 47 such threads all running at once. If the same misses take 250 ns because they go to CXL, each thread moves only about 2.6 GB/s, so you need proportionally more requests in flight to fill the same pipe. That is why bandwidth-bound workloads tend to be highly parallel or streaming ones, and why a single-threaded pointer chase cannot use extra bandwidth no matter where its pages sit.

Why a slower node can raise throughput. The gain exists only while the CXL side still has headroom.

How much bandwidth is on offer? In the Demystifying CXL setup, two local DDR5 channels provided roughly 2 times the CXL device’s bandwidth for stores and roughly 3.4 times for loads. Counted as raw capacity, the CXL link therefore adds somewhere between 30% and 50% to what the two channels can deliver. The authors note that a future socket with eight DDR5 channels and four CXL devices would keep the same ratio. Yet DLRM’s measured optimum put 63% of its pages on CXL, far more than a bandwidth-proportional split of those two pipes would suggest. The paper’s own analysis points at queueing at the DDR controller, threads spilling across sub-NUMA nodes and last-level cache effects, and the authors motivate their run-time tuner by noting that the right share depends on the device’s bandwidth and on what else is running.

Weighted interleave, and why the weights matter

Plain interleave is the obvious first tool, and it is the wrong one. MPOL_INTERLEAVE, covered in Placement Policy as an API, deals pages round-robin across the nodes in its mask, so DDR and CXL each receive half. That equal split assumes the nodes are equally capable. On a machine where the CXL link carries a third of what DDR can, half the traffic saturates the link long before DDR is busy. Sun and colleagues found exactly this: naively allocating 50% of pages to CXL, as the default interleave does, may produce lower throughput than keeping every page in DDR, even with more aggregate bandwidth available.

Weighted interleave fixes the ratio. Added in Linux 6.9, MPOL_WEIGHTED_INTERLEAVE spreads pages across the policy’s nodes according to per-node weights, and the set_mempolicy(2) manual page gives the example of nodes weighted 4, 7 and 9 receiving pages in a 4:7:9 ratio. The numactl tool exposes the same policy with its -w option, so you can try it on an unmodified binary. On a machine whose DDR is node 0 and whose CXL memory is node 2, an experiment might look like this:

cat /sys/kernel/mm/mempolicy/weighted_interleave/node0 /sys/kernel/mm/mempolicy/weighted_interleave/node2
echo 8 > /sys/kernel/mm/mempolicy/weighted_interleave/node0
echo 1 > /sys/kernel/mm/mempolicy/weighted_interleave/node2
numactl -w 0,2 ./your-service

If you run that, remember that the weights are global for the machine, not private to your process, so another tenant using weighted interleave inherits your choice. We would treat them like any other host-level tuning knob, owned by whoever owns the box.

The benefit of getting the ratio right is large. When LWN covered the original weighted-interleave patches in 2023, it quoted Gregory Price reporting a 165% increase in bandwidth utilisation compared with 1:1 interleaving between DDR and CXL, with 85% of the bandwidth allocated to DDR and 15% to CXL, measured with Intel’s Memory Latency Checker. A year later, in a session on extending the memory-policy interface, Price described a streaming benchmark on a single-socket machine with attached CXL memory running 78% slower than on a DRAM-only machine. With a proper task-wide weighted-interleave policy, the same benchmark landed between 6% slower and 4% faster, which he called better, “but it still sucks”, because processes have to be NUMA-aware to get there.

One allocation stream under three policies. Only the weighted split is shaped like the hardware, and only when the weights are right.

Where should the weights come from? At the 2025 memory-management summit, Joshua Hahn argued that weight ratios should roughly follow the bandwidth ratios of the controllers involved, and that weights should be small numbers, since a weight is simply how many pages go to a node before the allocator moves on. As an illustration, suppose a socket has eight DDR5-4800 channels at 38.4 GB/s each, about 307 GB/s in total, and one CXL device that roughly matches a single such channel. The bandwidth ratio is about 8:1, so weights of 8 and 1 would place roughly one page in nine, about 11%, on the CXL node.

The kernel can now do that arithmetic itself. Its ABI documentation for weighted interleave describes an auto mode, in which node weights are recalculated from bandwidth data whenever it becomes available at boot or on hot-plug, and a manual mode in which only you change them. Writing any weight by hand switches the system to manual mode. Two further details from the same page matter in practice. Weights range from 1 to 255, and they affect only new allocations, so changing them never migrates pages already placed.

Two cautions sit underneath every weight. First, interleave works at page granularity, and the set_mempolicy(2) page notes that accesses to any single page are still limited to one node’s bandwidth, so the gain is statistical across many pages. Second, a weight derived from peak bandwidth is a starting point, not an answer. The DLRM optimum moved with thread count, and Sun and colleagues observe that a lower-bandwidth device would want a smaller share of pages, so we should expect the best ratio to depend on the device, the workload and whatever else is running.

Why does Hahn insist on small numbers? With weights of 8 and 1, the allocator places eight consecutive pages on DDR and then one on CXL. Weights of 80 and 10 give the identical ratio, but every run is ten times longer, so a modest buffer of a few hundred kilobytes could land entirely on one node and never see the split at all. We would therefore reduce weights to their smallest integer ratio before deploying them, and accept a slightly rounded ratio over a precise but coarse one.

Bandwidth-bound or latency-bound? Find out first

Everything above assumes the workload is short of bandwidth. If it is short of latency instead, every page sent to CXL is a loss, as Redis showed. So the first job is to find out which you are, and the answer is rarely visible from the code.

Robey and Zamora give the vocabulary. A workload’s arithmetic intensity is the number of floating-point operations it performs per byte or word moved from memory, while a machine’s balance is its peak compute divided by its memory bandwidth. Processors have gained compute far faster than bandwidth, and they note machine balances of around 50 flops for every 8-byte word, whereas most applications run at roughly one flop per word. On a roofline plot, that puts most code on the sloped, bandwidth-limited part of the roof, well short of the compute ceiling.

Being on the sloped roof, though, is not the same as being bandwidth-bound in practice. A pointer-chasing service may move few bytes yet still stall on every miss, because it cannot keep enough requests in flight to fill the pipe. The test that matters is how close your achieved bandwidth sits to what the machine can deliver. The STREAM benchmark, which Robey and Zamora use to measure that ceiling, gives you the denominator; a memory-bandwidth counter during a representative load gives you the numerator. The cheap rungs of the evidence ladder in Proving You Have a NUMA Problem apply unchanged here. Sun and colleagues sampled their counters with Intel’s PCM tools, and they found a second signal worth borrowing: L1-miss latency, which rises both when a workload is cache-unfriendly and when the memory controller is queueing. If your miss latency climbs as load rises while your bandwidth plateaus, you are probably queueing, and that is the regime where spreading helps.

The most direct experiment is also the cheapest. Run the same load twice, once with numactl --membind on the DDR node and once with -w across DDR and CXL at a modest weight, and compare throughput and p99 latency. If throughput rises and the tail holds, we have our answer. If both get worse, the workload was waiting for trips rather than for a turn at the controller, and we should leave its pages where they were. Either way the experiment costs an hour and no code changes, which is a large part of why weighted interleave is worth knowing about.

A roofline-style triage. The 40% and 60% lines are the thresholds Meta’s study used for its production fleet, not universal constants.

The thresholds come with a caveat worth keeping. Meta’s figures are one-minute fleet averages, and the authors point out that bandwidth-sensitive services can become badly bandwidth-bound for shorter intervals, which shows up as tail spikes rather than as a high average. The SURGE study by Kadiyala and Daglis adds the other boundary: workloads whose demand stayed at or below half of the primary memory’s peak saw little or no benefit from extra bandwidth, because they were not queueing in the first place.

Where spreading wins

The clearest wins share a profile: streaming access, many requests in flight, and a working set far too large for caches to absorb.

Recommendation embeddings are the canonical case, and DLRM’s 88% gain is the headline number. Embedding reduction typically gathers rows from tables far larger than any cache, from many threads at once, which gives it exactly the many-requests-in-flight profile the identity above calls for; Sun and colleagues cite earlier work establishing that it is bound by memory bandwidth. Sun and colleagues also built Caption, a runtime tuner that samples memory-subsystem counters, estimates performance, and adjusts the share of new pages sent to CXL through the memory-policy interface. For four SPEC benchmarks, fotonik3d, mcf, roms and cactuBSSN, it found splits that beat the best static policy by 19%, 18%, 8% and 20% respectively, settling at 29% to 41% of pages on CXL. For DLRM it reached 80% more throughput than all-DDR, while landing 4% short of a plain 50:50 split, a reminder that a search is not guaranteed to find the peak.

Long-context model training provides a subtler example. Liaw and Chen studied CXL-attached memory for LLM fine-tuning with CPU offloading, where the optimiser runs on the CPU over parameters held in system memory. Naively interleaving everything across DRAM and CXL hurt badly: the optimiser step slowed by about four times once its data passed roughly 20 million elements, because its read-modify-write loops sit on the critical path. Their fix was to split by data rather than by process. Latency-critical optimiser state stays pinned in DRAM, while latency-tolerant tensors feeding GPU transfers are striped across one or more CXL add-in cards. With one card they recovered 97% to 99% of DRAM-only throughput, with two they matched it, and they beat naive interleaving by up to 21%.

Placement by data, not by process, in the fine-tuning study. The win came from separating the two halves, not from interleaving more.

SURGE shows that the exact split matters less than you might fear. Its derived traffic split landed close to the measured optimum, and a wide plateau around the peak kept performance within 10% of optimal across a range of splits. Across its workloads it cut average memory queueing delay by 36% and delivered an average speedup of 1.1 times, up to 1.3 times. Notably, the authors found that weighted first-touch placement was sufficient for their deployments. Getting into the right neighbourhood is most of the value.

What the wins have in common is predictable from the identity earlier in this part. Each workload keeps many requests in flight, from many threads or from deep prefetching of streaming data, so it can absorb CXL’s longer latency without leaving the link idle. Each also touches far more data than the caches hold, so the DDR controllers really are the bottleneck. When we look at a new workload, those two properties are the first things we would check, before reaching for any policy.

Where it is a trap

The traps mirror the wins, and each one is a way of applying the bandwidth model to something that does not fit it.

The first trap is a latency-bound workload. Redis never saturated its memory bandwidth, so spreading its pages simply added CXL latency to its critical path, and throughput fell in proportion to the share moved. Nothing about weights can fix that, since there was no queue to relieve.

No ratio rescues a workload that was never waiting in line. Caption, notably, learned this on its own: it kept Redis mostly on DDR, ending 3.2% ahead of a 50:50 split yet still 8.6% behind all-DDR.

The second trap is a process-wide policy over mixed data. Weighted interleave applies to everything the policy covers, so if you set it with numactl on a whole process, the small, hot, latency-sensitive structures get spread along with the bulk data. The fine-tuning study is the measured version of the warning in the placement-policy article: the optimiser step paid fourfold for being interleaved with tensors that wanted bandwidth. If the two halves can be separated by address range, they should be.

The third trap is assuming a fixed ratio stays right. Weights are global and apply to new allocations, whereas the best split moves with thread count, co-running tenants and device health. At the 2025 summit Hahn himself doubted that bandwidth use was the right metric to optimise, and the session ended with a general consensus that a better interface is needed; Michal Hocko had observed that the NUMA interface was designed for static hardware and falls short even on current systems. If your load shifts through the day, a ratio tuned at noon may be wrong by evening.

The fourth trap is quieter, and it comes from combining this article with the previous one. Weighted interleave places hot pages on the CXL node on purpose. Tiering-mode NUMA balancing, if you have it enabled, treats hot pages on a slow node as mistakes to repair, and promotes them back to DRAM. The two mechanisms may therefore work against each other on the same pages, one spreading and the other gathering, with migrations as the only visible output. LWN’s report on the memory-policy discussion draws the underlying distinction cleanly: tiering is about moving memory, while memory policy is about where it is allocated in the first place. We would pick one per workload rather than stack both.

The fifth trap is about capacity. Pages sent to the CXL node for bandwidth occupy CXL capacity, which is the same capacity the tiering article wanted for cold overflow. On a machine that relies on CXL to hold a large cold set, interleaving a hot working set onto it may leave less room than you planned for, and the kernel will not warn you. If you use CXL for both purposes, you need to budget its gigabytes as well as its gigabytes per second.

Choosing a policy

The decision is short once the measurement is done, and most of its branches end with not spreading.

A policy decision tree for CXL bandwidth. Separation by address range comes before any ratio.

If you do spread, start from the bandwidth ratio and measure around it, rather than trusting a single number. The plateau that SURGE observed suggests you will rarely need precision, but the DLRM result suggests the optimum can sit well away from the naive ratio, so a short sweep of splits under realistic load is worth the afternoon it costs. In practice we would step the CXL share in increments of ten percentage points, from zero to about half, recording throughput and p99 latency at each step under the load we actually care about. The curve tells you where the plateau starts and, just as usefully, where the tail starts to pay for it. Watch the tail as well as the throughput while you do, since a gain in requests per second that arrives with a worse p99 may not be a gain your users can see.

The two workloads, re-read

Go back to the opening pair. DLRM got 88% faster because its bottleneck was a queue at the DDR controllers, and sending pages to the slower node shortened that queue. Redis got slower because its bottleneck was the length of each trip, and the same move made trips longer. The hardware and the policy knob were identical; the difference was entirely in which resource each workload was short of. If you keep one diagnostic, make it the question we started with: are you waiting in a queue, or waiting for a trip?

That is why capacity is the boring half, at least for us as software engineers. Treating CXL as extra gigabytes needs nothing more than the tiering machinery from the previous article and some patience. Treating it as extra bandwidth asks us to measure our workloads, to place data by its access pattern, and to accept that the right answer sometimes looks like a mistake on a latency chart. We have now covered both halves on a single machine. The next article turns from one machine to many: when CXL memory sits in a shared pool, the question becomes who owns which gigabyte, and what the pool really saves.

References

  • Sun, Y., et al. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices. MICRO ‘23. — DLRM embedding reduction (63% on CXL, 88% higher throughput); Redis throughput losses by share on CXL; naive 50% allocation losing to all-DDR; the two-channel DDR to CXL bandwidth ratio; queueing at the memory controller; Caption’s design and results

  • Liaw, Y.-C., & Chen, S.-H. Analysis and Optimized CXL-Attached Memory Allocation for Long-Context LLM Fine-Tuning. — optimiser slowdown under naive interleaving; tensor-level placement; 97–99% and ~100% of DRAM-only; up to 21% over naive interleave

  • Kadiyala, D. K., & Daglis, A. Pushing the Memory Bandwidth Wall with CXL-enabled Idle I/O Bandwidth Harvesting (SURGE). — no benefit at low demand; 36% less queueing, 1.1x average and 1.3x peak speedup; the plateau around the optimal split; weighted first-touch placement

  • Li, H., Berger, D. S., et al. Pond: CXL-Based Memory Pooling Systems for Cloud Platforms. ASPLOS ‘23. — a x8 CXL port roughly matching a DDR5-4800 channel

  • Workload Behavior Driven Memory Subsystem Design for Hyperscale (Meta). — the 60% and 40–60% bandwidth-utilisation classes and transient peaks

  • Das Sharma, D., Blankenship, R., & Berger, D. S. An Introduction to the Compute Express Link (CXL) Interconnect. — loaded latency and CXL’s bandwidth advantage

  • Robey, R., & Zamora, Y. Parallel and High Performance Computing. Manning, 2021. — arithmetic intensity, machine balance, the roofline, the STREAM kernels, and bandwidth as the common limit

  • LWN, Weighted interleaving for memory tiering (2023); Extending the mempolicy interface for heterogeneous systems (2024); Automatic tuning for weighted interleaving (2025). — Price’s 165% result; the streaming benchmark; weights near bandwidth ratios; the limits of today’s NUMA interface; tiering versus allocation

  • Linux kernel documentation, NUMA Memory Policy. — weighted interleave and its sysfs weights

Engineering Deep Dives goes beyond tutorials — covering system design, performance, and the “why” behind engineering decisions. Upgrade to get every post in full.

© 2026 Shriom Tripathi
548 Market Street PMB 72296, San Francisco, CA 94104