mailwal

CXL Memory Pooling: Stranded Memory and Real Savings

Engineering Deep Dives

Welcome to Engineering Deep Dives — deep, practical content for engineers who want to go beyond the surface.

CXL Memory Pooling: Stranded Memory and Real Savings

CXL memory pooling is a bin-packing fix for stranded DRAM, not a speedup. How Pond pools memory, what MLDs and fabric managers do, and where it stops paying.

Stranded Memory and the Economics of Pooling

Look at the memory bill for a public cloud and one number stands out before any other. DRAM can account for half the cost of a server, according to figures cited by the authors of Pond, Microsoft’s CXL pooling design. Their analysis of Azure traces from 100 production clusters then shows how much of that memory does nothing. At the 95th percentile, between 3% and 27% of memory in a cluster was stranded, with outliers reaching 36%. Separately, about half of all virtual machines touched less than half of the memory they had rented.

That memory was bought, installed, powered and cooled. Some of it could not be rented to anyone, and some of it was rented to a customer who never used it. Neither problem is a performance problem, and neither is solved by faster memory.

The core insight: CXL memory pooling is not a performance feature. It is a bin-packing fix for memory that was bought and never used. Every pooled gigabyte is slower than the local gigabyte it replaces, so the whole case rests on three questions: how much memory the scheduler strands, how predictably workloads leave memory untouched, and whether the latency and failure costs of a shared device stay smaller than the money it saves.

We have spent four articles on one machine, first as capacity and then as bandwidth. This one steps back to the fleet, where the question changes from “where should this page live?” to “which server should own this gigabyte today?”

Stranding is a scheduling artifact

Pond’s authors define the term precisely. Memory is stranded when all the cores of a server are rented out but unallocated memory remains, and nobody can rent it, because every VM also needs cores. Das Sharma, Blankenship and Berger give the general version in their CXL introduction: a resource is stranded when idle capacity remains while another resource is fully used, and the underlying cause is that compute, memory and I/O devices each belong to exactly one server. So every server is provisioned for peak demand, and the leftovers cannot move.

Nothing about the hardware makes memory strand; the scheduler does, though for good reasons.

The Kubernetes documentation on resource management describes the same mechanism in a setting you may know better: the scheduler ensures that, for each resource, the sum of requests on a node stays below its capacity, and it refuses a placement when that check fails even if actual usage is low. That rule is correct, since it protects nodes from tomorrow’s peak. Yet it means a node whose CPU requests are exhausted cannot accept a pod that would fit comfortably in its spare memory.

An illustrative pair of servers. Each looks full from one angle and half empty from the other, and the scheduler cannot combine them.

Operating systems students meet this problem under another name. Silberschatz, Galvin and Gagne describe external fragmentation as having enough total free memory to satisfy a request while no single hole is large enough, and the 50-percent rule that comes with first-fit allocation: for every N allocated blocks, about 0.5N more are lost to fragmentation, so roughly a third of memory may be unusable. Stranding is external fragmentation at the granularity of servers. The free memory is real; it is simply in the wrong box.

The Azure data also shows how stranding behaves over time, which matters for anyone hoping to plan around it. Stranding rose with utilisation, from about 6% of memory in clusters where 75% of cores were scheduled to over 10% at around 85%. At the 95th percentile it reached 25% during busy periods, with individual outliers at 30%. A workload change could raise it suddenly across many racks at once, and the authors found it generally hard to predict which clusters or racks would strand memory next. If you cannot predict where the waste will appear, you cannot fix it by buying a different server shape.

You can, however, measure how much of it you have, and on Kubernetes the data is already there. The documentation shows kubectl describe nodes reporting each node’s allocatable capacity and the CPU and memory requests allocated against it. If you collect those two percentages for every node, the nodes whose CPU requests sit near 100% while their memory requests sit well below are stranding memory, and the ones in the opposite corner are stranding cores. We would plot that scatter before reading any vendor’s pooling deck, because it tells you whether the problem exists in your fleet at all.

Untouched memory, the second leak

Stranding is memory that nobody rents. The second leak is memory that someone rents and never uses.

Pond’s telemetry on opaque VMs found that the median VM left 50% of its memory untouched, and that even in the cluster with the least untouched memory, over half of the VMs left more than 20% of theirs unused.

The cause is again the contract rather than the hardware. Brendan Burns, Joe Beda and Kelsey Hightower point out in Kubernetes: Up and Running that a request specifies a minimum, not a maximum, and that the scheduler guarantees the requested amount by reserving it. A VM size works the same way. Customers buy a shape with headroom, the provider reserves the whole shape, and the headroom sits idle because nobody, quite reasonably, wants to run out.

If you run a platform team, you have probably seen the same leak as the gap between requested and used memory on your dashboards. It tends to persist, though, because shrinking a request is a bet that the service will never need the headroom, whereas the cost of losing that bet is a node under memory pressure and a pod evicted or killed. Pooling offers a third option: keep the request, but back the part you expect to stay untouched with memory that is cheaper to waste.

Untouched memory is also a much better target than stranded memory, although the reason only becomes clear in a moment. A gigabyte that a workload never touches costs nothing in performance wherever it lives. If we could predict which gigabytes a VM will never use, we could back exactly those with slower, shared memory and nobody would notice.

CXL memory pooling: from dedicated memory to a pool

Pooling attacks both leaks by decoupling memory from servers. Instead of fitting every server with enough DRAM for its worst case, we give each server a smaller local share and connect a group of servers to a shared memory device. The CXL introduction frames the payoff in one sentence: instead of sizing hosts for the worst-case memory any workload might need, operators can provision memory for the average case.

Pond’s hardware is an external memory controller, an ASIC with several x8 CXL ports and DDR5 channels behind them, wired directly to a group of CPU sockets. Each host maps the controller’s whole capacity at boot but treats it as offline. A pool manager then assigns memory in 1 GB slices, each to at most one host at a time, and the controller checks every access against an ownership table, so a host reading a slice it does not own takes a fatal memory error. Hosts keep separate cache-coherence domains. Nothing is shared at any moment; capacity is only reassigned over time.

Dedicated against pooled provisioning. The capacities are illustrative; the shape is Pond’s: small pools, direct links, one owner per slice.

Pool size is the first economic dial we can turn, and it has diminishing returns. In Pond’s analysis, pooling across 16 to 32 sockets could cut cluster memory demand by about 10%. With half of each VM’s memory in the pool, 32 sockets saved 12% of DRAM and 64 sockets saved 13%. Larger pools cost latency, though. The authors estimate that CXL adds 70 to 90 ns over same-NUMA-node DRAM for pools of 8 to 16 sockets and more than 180 ns at rack scale, because larger pools need switches, which add at least 70 ns each, and retimers, which add about 10 ns in each direction. Their multi-headed controller cut latency by about a third compared with a switch-only design. Their conclusion, that 8 to 16 sockets capture most of the benefit, is a direct trade between the savings curve and the latency ladder from the first article.

Put a number on that trade. Suppose a 16-socket pool group where each socket would otherwise carry 512 GB, 8 TB in total. A 7% reduction in DRAM, Pond’s figure for such a pool, removes roughly 570 GB. That is about one socket’s worth of memory saved per 16, which is real money when multiplied across a fleet, yet small enough that the pool’s controllers, cables and control plane must cost much less than the DRAM they replace. Whether they do depends on prices you will know better than any paper.

Pond’s trick: put the pool where nobody looks

A pool by itself, however, would not survive contact with customers.

Pond’s own measurements show why: with pool latency of 64 or 140 ns over local memory, only 43% and 37% of 158 workloads stayed within 5% of local performance when running entirely from the pool, and more than 21% lost over a quarter of their performance. Backing whole VMs with pooled memory would break service-level promises for a large minority of them.

Pond’s answer combines the two leaks with the CPU-less node from A NUMA Node With No CPUs. The hypervisor exposes pool memory to a VM’s guest as a zero-core virtual NUMA node, which the authors call zNUMA. Guest operating systems prefer local memory and fall back to a CPU-less node only when local memory runs out, so if the zNUMA node is sized to match the memory the VM will never touch, it is never used. Here the first-touch behaviour that surprised us on a CXL node becomes the feature that makes pooling safe.

The hard part is predicting how much memory a VM will leave untouched. Pond uses two machine-learning models fed by telemetry the hypervisor already has. The first predicts whether a VM is insensitive to memory latency, using core performance counters such as the memory-bound share of pipeline stalls; insensitive VMs go entirely into the pool. The second predicts untouched memory from VM metadata, most importantly the percentiles of untouched memory in the same customer’s recent VMs. Everything else gets local DRAM plus a pool share equal to its predicted untouched memory, rounded down to whole gigabytes.

Notice how this reframes the problem we met in the tiering article. There, the classifier had to predict which pages would be hot in the next few minutes, and it was often wrong. Here, the prediction is coarser and arguably easier: roughly how much of a VM’s memory will stay untouched for its whole life, based on what the same customer’s VMs did last week. A coarse, stable prediction made once at placement time is likely easier to get right than a fine-grained one repeated every few seconds.

Pond’s allocation and correction loop. Every arrow out of a prediction has a cheap fallback.

What makes the design trustworthy is that wrong guesses are cheap. If Pond underestimates untouched memory, the VM simply never reaches the pool. If it overestimates, a quality-of-service monitor watches the same counters, and when a VM’s slowdown exceeds the configured margin, Pond makes a one-time correction that copies the VM’s memory to local DRAM, at about 50 ms per gigabyte. Memory always stays mapped in the hypervisor’s page tables, so even a bad prediction degrades performance gradually rather than falling off a cliff.

The results are modest, but that is the point.

At a pool of 16 sockets, Pond cut overall DRAM requirements by 9% when pool latency was 182% of local and by 7% at 222%, while a strawman that gave every VM a fixed 15% of pool memory saved only 3%. The authors configure the system to keep performance within 1% to 5% of same-NUMA-node allocation. With DRAM at roughly half of server cost, a 7% DRAM saving becomes about 3.5% of total server cost, which they describe as hundreds of millions of dollars for a large cloud provider. For an organisation with fifty servers, the same 3.5% would buy very little, and probably not the engineering it takes to earn it.

MLDs, dynamic capacity and the fabric manager

Pond built its own controller and a private management path, but the CXL standard defines the building blocks for doing the same with commodity parts. The vocabulary is worth knowing, because it tells you who is in charge of which gigabyte.

CXL 2.0 adds switching and pooling. Each host sees its own virtual hierarchy through the switch, and a standard device, a single logical device, belongs to one host at a time. A multi-logical device goes further: it can divide its memory into up to 16 logical devices and assign each to a different host simultaneously. The switch tags each host’s traffic with the right logical-device identifier, so the hosts never see the partitioning. CXL 3.0 then adds the dynamic capacity device, which the kernel’s CXL device documentation describes as offering an allocator-like interface for physical memory to a fabric manager, handing out memory extents that may be exclusive to one host or shared across several.

How a multi-logical device is partitioned and bound. The hosts see ordinary hot-plugged memory; the fabric manager sees the whole map.

The fabric manager is the policy brain. The CXL introduction notes that it can be software on a host, firmware in a baseboard management controller or a switch, or a dedicated device, talking to the hardware in-band or over an out-of-band management protocol. To assign memory it issues a bind command naming the switch, the virtual bridge, the port and the logical device. The switch then sends a hot-add indication to the host. To take memory back it issues an unbind, with options that range from waiting politely for the host to release the port to forcing a hot-remove, which exists precisely for uncooperative hosts such as a malfunctioning server or a bare-metal tenant.

That last option deserves a moment of attention. A pool shared between tenants is a place where one party can take memory away from another, and the protocol plans for the case where the other party does not agree. If you run a pool, the fabric manager is part of your trust boundary, and its decisions need the same scrutiny as your scheduler’s.

Linux support, however, is still catching up with the specification. The kernel’s documentation states that, as of v6.14, Linux provides no formalised interface for managing dynamic capacity devices, nor for multi-headed devices without dynamic capacity, though work is under way. Pond, for its part, drove slice assignment with its own driver interrupts and noted dynamic capacity as a future alternative.

If you are evaluating pooled-memory hardware, those gaps translate into questions for the vendor. Who runs the fabric manager, and what happens to bound memory if it restarts? Which of the bind and unbind options does it use, and will it ever force a hot-remove on a running host? Unless the answers fit the kernel version you actually deploy, you may find yourself writing the management layer that Pond’s authors wrote for themselves.

Hot-plug, seen from the host

Every pool reassignment is a memory hot-plug event on some host, and the host side has rules of its own. The kernel’s CXL device hot-plug notes say firmware must reserve resources at boot, proximity domains and physical address windows, for any device that might arrive later. Hot-add is bounded by the window reserved at boot, and some platforms omit the window entirely when no CXL device is present, in which case hot-add is impossible. Removal is harsher still: pulling a CXL memory device without first tearing down the software that manages it is likely to machine-check the system, or at least deliver SIGBUS to any user-space process touching it.

Pond’s measurements show the asymmetry in practice. Onlining a slice took microseconds per gigabyte, while offlining took 10 to 100 milliseconds per gigabyte, because the memory must be emptied first. Offlining also needs the whole contiguous gigabyte to be free, so Pond allocates VM memory in aligned gigabyte units and confines host agents and drivers to a separate host-local partition, which keeps kernel allocations from fragmenting the pool. This is the ZONE_MOVABLE argument from the second article and the kernel’s memory hot-plug guide, at fleet scale: memory you may want back must never hold anything you cannot move.

Granting pool memory is fast and reclaiming it is slow. Pond keeps a reserve so that the slow half never sits in front of a VM start.

What pooling asks of a cluster scheduler

Pooling turns memory into what we might call a two-level resource. A scheduler can no longer treat a server’s memory as one number; it has local memory with one latency, pool memory with another, and a pool that a fixed group of neighbours shares. Placement becomes constrained to that neighbourhood, because a VM can only borrow from the pool its host is wired to.

The scheduler also has to predict, not just count. Pond’s scheduler asks a prediction service how much local memory a VM needs, tells the pool manager which host needs which slices, and starts the VM once onlining completes. Since offlining is too slow for the critical path of a VM start, Pond keeps a buffer of unassigned pool memory that is refilled asynchronously as VMs leave. Without that buffer, the slow half of hot-plug would sit in front of every placement.

In Kubernetes terms, the gap between a pod’s memory request and what it actually touches is the zNUMA candidate. Nothing in today’s request-and-limit model expresses “this much of my request may live in slower, shared memory”, so a pooled Kubernetes cluster would need either a new resource class or a node-level component that makes the same predictions Pond’s hypervisor does. Until one exists, we would treat pooling as an infrastructure decision made below the orchestrator, not as something application teams configure.

When you design such a layer, the asymmetry of hot-plug should drive its shape. Adding memory to a host is fast and safe, whereas taking it back is slow and can fail if anything unmovable has landed on it. A pool controller should therefore grant memory eagerly from a reserve and reclaim it lazily in the background, which is precisely what Pond’s buffer does.

The honest limits

Pooling widens the blast radius of memory, and every design decision in Pond is, in part, an answer to that rather than an afterthought. Hosts never interleave across pool memory, only across local memory, so a failed memory controller affects only the VMs that had slices on it while VMs using other controllers carry on. A host failure is isolated, and its slices return to the pool. Even a pool manager failure stops reassignment without touching the data path. The CXL protocol helps from its side with a containment model, in which an unresponsive device produces error responses instead of hanging the host, and with standardised error reporting that includes poison. Yet the CXL introduction itself lists a rigorous approach to error containment and to CXL’s increased blast radius as open work.

A memory pool is a shared dependency, and its failure modes are still being written down.

Failure domains in a Pond-style pool. Each limit on the blast radius comes from a deliberate design choice, not from CXL itself.

Noisy neighbours are the second limit, and they are subtler. A multi-logical device and its switch are shared, so heavy traffic from one host can congest the others. The CXL introduction describes the countermeasures: a DevLoad field in responses that tells hosts to throttle their request rate under load, source throttling that caps how much of a device’s resources each host may consume, and fabric-manager controls for bandwidth allocation among logical devices. All of them work, but each is one more knob that someone has to own.

The third limit is the size of the prize, and we should be honest about it. Seven percent of DRAM is large for a hyperscaler and small for nearly anyone else, and it arrived with machine-learning models, a control plane, custom hardware and a monitoring system to catch the models’ mistakes. If your fleet is small, your workloads are homogeneous, or your memory is already well utilised, the cheaper fix may be the oldest one: change the server shapes or the request sizes so that cores and memory run out at roughly the same time.

What pooling does not do for GPU clusters

One claim about pooling deserves a direct correction, because it circulates widely. It says that a CXL memory pool gives GPUs a shared, coherent memory, so that collective operations such as AllReduce no longer need to stage and synchronise data. Pooling, as we have seen, does the opposite of sharing. Each slice belongs to one host at a time, hosts keep separate coherence domains, and reassignment happens over seconds, not within a training step. Coherent sharing across hosts is a CXL 3.x feature with costs of its own, and it is the subject of the next article.

The GPU side of the claim does not hold up either. The survey by Chen and colleagues notes that commercial GPUs do not support the CXL standard, so a GPU’s requests to CXL memory must be translated by the CPU, and in that path the PCIe link to the GPU becomes the main bottleneck. Building a unified memory with direct CXL access requires GPUs that act as CXL Type 2 devices, and the work the survey describes on that front is modelling, simulation and research prototypes.

None of that makes pooling irrelevant to GPU servers. It makes pooling relevant in a narrower, more honest way: as host-memory capacity for CPU offloading, where, as Capacity Is the Boring Half showed, careful placement on CXL cards recovered 97% to 99% of DRAM-only throughput for long-context fine-tuning.

The bill, re-read

Let us go back to the opening numbers. Up to a quarter of a busy cluster’s memory stranded by the scheduler, and half of each VM’s memory untouched by the guest. Pooling recovers part of that waste, about 7% of DRAM in Pond’s case, by moving gigabytes between servers and by hiding the untouched ones in a node nobody’s first touch ever reaches. It does not make anything faster, and it charges for its savings in latency, in a wider failure domain, and in a control plane that must predict behaviour it cannot see.

So the question we would ask of any pooling proposal is not how fast the pool is. Ask how much memory your scheduler strands today, how predictably your workloads leave memory untouched, and who will own the fabric manager when it takes memory away from someone at three in the morning. If the first two numbers are small, the cheapest pool is a better server shape.

References

  • Li, H., Berger, D. S., et al. Pond: CXL-Based Memory Pooling Systems for Cloud Platforms. ASPLOS ‘23. — stranding and untouched-memory measurements; pool-size savings; latency by pool size, switches and retimers; the multi-headed controller and slice ownership; zNUMA; prediction models, telemetry and the QoS monitor; onlining and offlining costs; failure management; DRAM and server-cost results

  • Das Sharma, D., Blankenship, R., & Berger, D. S. An Introduction to the Compute Express Link (CXL) Interconnect. — stranding as a coupling problem; provisioning for the average case; CXL 2.0 pooling, virtual hierarchies and single-logical devices; multi-logical devices; DevLoad, source throttling and containment; fabric manager placement, bind, Set-LD and unbind; global fabric-attached memory; blast radius as open work

  • Chen, C., et al. Next-Gen Computing Systems with Compute Express Link: a Comprehensive Survey. — DirectCXL pooling prototypes; GPUs reaching CXL memory through the CPU; unified memory requiring Type 2 GPUs

  • Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed. — external fragmentation and the 50-percent rule

  • Burns, B., Beda, J., & Hightower, K. Kubernetes: Up and Running, 2nd ed. — requests as a reserved minimum, not a cap

  • Liaw, Y.-C., & Chen, S.-H. Analysis and Optimized CXL-Attached Memory Allocation for Long-Context LLM Fine-Tuning. — CXL cards as host memory for CPU offloading

Engineering Deep Dives goes beyond tutorials — covering system design, performance, and the “why” behind engineering decisions. Upgrade to get every post in full.

© 2026 Shriom Tripathi
548 Market Street PMB 72296, San Francisco, CA 94104