mailwal

CXL Shared Memory: Coherent, but Not One Big Machine

Engineering Deep Dives

Welcome to Engineering Deep Dives — deep, practical content for engineers who want to go beyond the surface.

CXL Shared Memory: Coherent, but Not One Big Machine

CXL 3.x back-invalidate gives hosts hardware-coherent shared memory. Why that is not DSM reborn, what locks still cost, and which uses are real or fantasy.

Shared Memory, Again, But Coherent This Time

Distributed shared memory was one of the most seductive ideas in systems research. Give a cluster of machines one address space, let any process load and store anywhere, and the hard parts of distributed programming, messages, serialisation, consistency, would disappear behind ordinary memory operations. Tanenbaum describes how it worked. When a CPU touched a page it did not have, the operating system trapped the fault, found the machine holding the page, asked it to unmap the page and send it across the network, and restarted the instruction. To the program, the cluster looked like one machine with shared memory.

It never became the way we build distributed systems, though it was influential. The pages were too big, the network was too slow, and coherence was maintained by software one page fault at a time. When people now read that CXL 3.x lets several hosts share memory with hardware cache coherence, the obvious reading is that DSM is back with its main weakness fixed. The survey by Chen and colleagues says almost exactly that: CXL’s hardware coherence avoids the network-based coherence mechanism that was the main bottleneck of traditional DSM.

That reading is half right, and the wrong half is dangerous.

The core insight: CXL 3.x fixes the mechanism that sank the last generation of DSM, software coherence at page granularity over a network, by moving it into hardware at cache-line granularity. It does not fix what made DSM a distributed system: hosts that fail independently, writes that contend across a slower link, and shared state that nobody owns when something breaks. Coherent means the hardware agrees about the bytes. It does not mean the hosts agree about anything else.

The previous article was careful to say that pooling is not sharing: each slice of a pool belongs to one host at a time. This one is about the other case, where the same cache line can sit in several hosts’ caches at once, and about what that does and does not buy us. Having followed the series this far, you already have most of the tools: the CPU-less node, the latency ladder, and the coherence costs from the NUMA series. What you need now is a sense of which old lessons apply at a larger scale, and we will find that nearly all of them do.

How DSM worked, and why page granularity hurt

The mechanics of classic DSM explain both its appeal and its trouble. Each machine kept its own page tables. A load or store to a page not mapped locally raised a page fault, and instead of reading the page from disk, the operating system fetched it from another machine’s RAM. Tanenbaum’s summary is blunt: the operating system was simply satisfying page faults from remote RAM instead of from local disk.

The unit of sharing was arguably the core problem. Tanenbaum contrasts the two worlds directly: a multiprocessor’s caches move blocks of 32 or 64 bytes, whereas a DSM system has to move at least a page, because the memory management unit works in pages, and often several pages at once to amortise the network trip. Larger units invite false sharing. Two unrelated variables that happen to live on the same page force that page to bounce between the machines using them, and Tanenbaum notes that the larger the effective page, the more often this happens. Nothing analogous exists in ordinary virtual memory, and even a clever compiler can do little when two nodes use different elements of the same array.

Consistency, however, was handled in software as well. A common approach replicated pages read-only on every machine that read them and invalidated those copies when someone wanted to write. Every one of those steps was a page fault, a protocol message and a page-sized transfer. When you multiply that by contention, you get a system whose performance depends on page layout in ways no programmer can see.

What did that mean for an ordinary shared counter? Suppose two machines take turns incrementing a counter that sits on a 4 KB page. Under classic DSM, every handoff is a page fault on the machine that wants to write, a protocol exchange to revoke the other machine’s copy, and a 4 KB transfer across the network. Under CXL 3.x, the same handoff moves a 64-byte cache line, one sixty-fourth of the data, and the hardware does it without a fault or a kernel in the path. That is a large improvement, and it is easy to stop reading there. Notice, though, that the handoff still happens on every increment, and that it still crosses a link between machines. The mechanism got much cheaper, whereas the pattern that made it expensive is exactly the same.

What changed and what did not. The first three rows improved by orders of magnitude; the last row is the same.

Back-invalidate: what CXL 3.x actually adds

The key mechanism is a new pair of message channels. In earlier CXL versions, the host orchestrated coherence and a memory device could not snoop the host’s caches. Das Sharma, Blankenship and Berger explain that CXL 3.0 adds a back-invalidate flow to the memory protocol, in which the device can send a snoop request back to a host and receive a response. That turns the memory device into the place where coherence for its memory is tracked.

For shared memory across independent hosts, the device keeps a directory or snoop filter. The CXL introduction walks through an example. When host 1 reads a cache line, the directory records it as shared by host 1. When host 3 reads the same line, the directory adds host 3 as a sharer. When host 4 wants to write, the device sends back-invalidates to hosts 1 and 3, waits for both to confirm that they have dropped their copies, marks the line exclusive to host 4, and only then sends the data and ownership.

The back-invalidate flow from the CXL introduction’s example. Host 4’s write waits on every other sharer.

The directory itself can be modest, although there are trade-offs. The authors describe two bits per line for the states invalid, shared and exclusive, followed by a list of sharers whose format is up to the device. That list may name each host individually up to some number, then fall back to coarse groups of hosts, and a device may track lines in groups such as whole pages. Precision costs directory space. The survey reports AMD’s caution that, as shared memory grows, a precise snoop filter tracking every 64-byte line may become impractical. A coarser directory is cheaper, yet it invalidates more hosts than strictly necessary, and every unnecessary invalidation is traffic on a link we already know to be slower than local memory.

What does a single write cost, then, when many hosts are reading? If you follow the example, the writer waits until every sharer has confirmed its invalidation, so the write is only as fast as the slowest host to answer. With a precise sharer list, that means every host that actually read the line. With a coarse list, it may mean every host in a group, including some that never touched it. Either way, the cost of a write grows with the number of readers, which is perhaps the single most important number to keep in mind when you design anything for this kind of memory.

Scale is also bounded differently from a network. The CXL introduction describes global fabric-attached memory devices that can serve up to 4,096 nodes. It also argues that the cost of cabling, retimers and length limits will probably keep CXL deployments within a rack or a few racks, with current financial models pointing to sub-rack deployments as the sweet spot. CXL is unlikely to replace Ethernet across a datacenter, by the same authors’ account, so shared memory is a rack-scale tool, not a fleet-scale one.

The coherence domain boundary

Each host, however, still has its own coherence domain. The CXL introduction is explicit that the back-invalidate flows implement hardware-coherent shared memory across multiple hosts, each with its independent coherency domain. Inside a host, the CPU’s caches, local DRAM, page tables and TLBs follow the host’s own protocol, exactly as before. The device’s directory extends coherence only over the shared region it owns, and only for the cache lines in it.

That boundary matters more than the coherence itself, since everything outside it is still private. The operating system on host 1 knows nothing of the scheduler, memory allocator, page cache or lock state on host 3. A pointer stored in shared memory that points into host 1’s local memory is meaningless on host 3. Even a pointer into shared memory is only valid if both hosts map the shared region at the same address, or if the data structure uses offsets rather than pointers. None of this is new; it is what every multi-process shared-memory program on one machine already deals with. What is new is that the processes now run under different kernels, with different clocks, and fail at different times.

If you design a structure for shared CXL memory, a few rules follow directly from the boundary. Store offsets from the start of the shared region rather than raw pointers, unless every host maps the region at the same virtual address. Never store a pointer into any host’s local memory. Allocate shared objects from an allocator that itself lives in shared memory and survives a crashed client, which is precisely the problem the CXL-SHM work set out to solve. And treat any metadata a host caches locally, a size, an index, a version, as something that can go stale without notice, because the hardware keeps only the shared lines coherent, not your private copies of what they meant.

Coherence crosses the boundary only through the shared region. Everything else about each host stays private, including how and when it fails.

Coherent is not the same as free

It’s Coherence, Not Distance showed what coherence costs inside one machine. When two cores write the same cache line, the line’s ownership travels back and forth between them, and that traffic, not the distance to memory, is what slows a contended counter or a busy lock. Tanenbaum gives the classic case. A test-and-set spinlock writes the lock word, so every attempt to acquire it pulls the whole cache block into the requester’s cache as an exclusive copy, invalidating the holder’s copy and shuttling the block back and forth.

Across hosts, the same pattern probably pays more for each round. Every transfer of exclusive ownership now goes through the device’s directory, and when other hosts hold copies, it waits for back-invalidates to reach each of them and return. We do not have published measurements of that latency on shipping CXL 3.x hardware. Yet the latency ladder from the first article gives a lower bound on its shape: a direct-attached CXL memory access already costs a few hundred nanoseconds, and an ownership transfer needs at least a request to the device plus a back-invalidate round trip to each other sharer.

There is one measured hint, though it comes from a single host rather than a shared fabric. In the Demystifying CXL study, Sun and colleagues found that ordinary stores reached 15 to 59 percent lower bandwidth efficiency than loads on their three CXL memory devices, and 74 percent lower on remote-socket DRAM. They attribute the gap to the implicit load behind each store and the cache coherence checks it triggers. So your writes already pay a coherence tax before a second host is involved at all; sharing adds the back-invalidates on top of it.

Lock cost by distance. The ordering follows from the mechanism; the absolute numbers for the bottom row are not yet published.

So the old advice holds with more force. Tanenbaum describes the standard remedies: read before you test-and-set, back off exponentially between attempts, or give each waiter its own private lock variable in a separate cache block, the Mellor-Crummey and Scott design. Every one of them reduces how often ownership moves. When you design a data structure for CXL shared memory, you should assume that each write to a line other hosts are reading is expensive, and arrange things so that hosts mostly write lines nobody else reads.

In practice, that usually means a few familiar patterns. Give each host its own slots and let readers combine them, rather than having every host update one shared counter. Publish data by writing a complete new version in lines nobody is reading yet, then flipping a single word to point at it, so readers see either the old version or the new one and writers contend only on that word. Batch small updates so that each ownership transfer carries more useful work. If you have tuned a lock-heavy service on a two-socket machine, none of this will surprise you; the difference is that the penalty for getting it wrong is larger and harder to observe from any one host.

The CXL introduction says as much in its own terms.

Hardware coherence can scale for applications where the shared data is mostly read-only, and in some cases software coherence may be preferable. That is a remarkable thing to read in a paper introducing the hardware coherence: it tells you the protocol was designed for read-mostly sharing, not for hosts fighting over the same lines.

The old trap: partial failure

Here is the very old trap, and we have met its shape before. Kleppmann draws the distinction that DSM blurred. A program on a single computer either works or does not; when hardware fails, the usual result is a total failure, such as a kernel panic. A distributed system has partial failures: some parts are broken while others keep working, nondeterministically, and you may not even know which. A shared-memory architecture, in his account, can treat all its components as a single machine, and it pays for that with limited fault tolerance.

CXL shared memory puts both models in one address space. The memory is shared like a multiprocessor’s, yet the hosts fail like a distributed system’s. Suppose host 1 takes a spinlock that lives in shared memory and then crashes, or simply pauses for a long garbage collection. The lock word still says “held”. There is no kernel spanning both hosts to notice the dead owner and release it, so host 3 spins forever, or until someone writes a timeout.

A crashed or paused holder leaves a lock and a half-written record behind. Leases and fencing tokens are the distributed-systems answer, and the data structure itself has to enforce them.

The distributed-systems answer is the one Kleppmann gives for locks in general. A lock should be a lease that expires, so that a crashed holder does not block everyone forever. Because a paused holder may wake up after its lease expired and still believe it owns the lock, every grant should carry a fencing token, a number that increases with each grant, and the protected resource must reject writes carrying an older token. His essay on distributed locking describes exactly this failure with a garbage-collection pause, and notes that stop-the-world pauses have sometimes lasted minutes. In shared memory, the “resource” is the data structure itself, so the token check has to live inside it.

What would you actually put in the lock word? At minimum, the owner’s identity, an expiry time and the current token, updated together with a single atomic operation on a line that ideally holds nothing else. The expiry needs margin, because hosts read their own clocks and those clocks are not perfectly aligned. Every write to the protected record should carry the token, and the record should keep the highest token it has accepted. That is more machinery than a spinlock, but it is the machinery that turns a crash from a permanent hang into a timeout and a retry, and we would not deploy write-shared structures across hosts without it.

Teams whose services already use leases and fencing through a lock service have most of the machinery they need; what changes is that the lock now lives in memory your own code manipulates directly. Researchers have started to treat this seriously, which suggests how open the problem still is. The survey reports that existing CXL shared memory lacks a failure model, and that cross-host sharing introduces new failure types, such as data lost while the process using it is still running, or a process that crashes while its data remains. CXL-SHM uses an era-based non-blocking reference-counting scheme so that memory management survives client crashes, and a UCSD group proposed a model that classifies data failures and process failures separately. Hardware adds one more mode. The famfs patch posting notes that poison or timeouts in shared memory can be expected to kill applications with SIGBUS, which is a failure an in-process lock library cannot catch on behalf of the other host.

What people actually build on CXL shared memory today

Look at the systems being built on shared CXL memory and a pattern appears: they lean hard on the easy cases, and some of them avoid hardware coherence altogether.

The kernel’s DAX notes state the baseline we have to start from. If capacity is shared across hosts, software must use appropriate flushing unless the region supports back-invalidate. Plenty of shared devices predate CXL 3.x, so software-managed coherence is the common case. If you are evaluating a shared-memory device, your first question should therefore be whether it supports back-invalidate at all, because your software design depends on the answer. The kernel’s CXL device notes describe dynamic-capacity extents that may be shared across hosts, and the famfs mode of daxctl from A NUMA Node With No CPUs exists for exactly this kind of memory. famfs, a filesystem for shareable memory from John Groves at Micron, is the clearest example. Its original patch posting says that famfs does not attempt to solve concurrency or coherency problems for applications, and names the two uses that are immediately useful: serial sharing, where one host at a time has access, and read-only concurrent sharing, where every host can cache freely. At LSFMM in 2024, Groves described the typical case as data written once and then read by many hosts, which needs no cache coherency at all. By the 2026 summit, famfs still had not been merged, and Groves remarked, to chuckles around the room, that CXL “makes cache coherency worse”.

RPC systems tell the same story from another angle. The survey describes HydraRPC, which marks its shared regions uncachable for reads and uses instructions that write straight to the device memory with fences, bypassing the caches entirely. CXL-SHM’s RPC framework passes object references rather than values, NotNets keeps payloads in shared memory and passes only references to them, and RPCool adds a way to transfer access permission over a payload so both sides cannot touch it at once, while keeping RDMA for traffic between racks. These designs are interesting precisely because they treat shared memory as a very fast message channel with explicit ownership handoffs, not as a coherent heap.

We find that pattern telling. The teams closest to the hardware are, for now, choosing to use less of its coherence rather than more, and to rebuild the explicit ownership that DSM tried to hide. That may change as CXL 3.x devices with back-invalidate become common. It is still a strong hint about where the risk lies.

The motivation for that channel is real, though narrower than it first looks. The CXL introduction notes that a 4 kB update takes under 2 microseconds to transmit at 50 GB/s, whereas communication delays on current networks exceed 10 microseconds, and that coherent shared memory could cut those delays to below a microsecond. For partition-and-aggregate services and consensus protocols with small updates, that gap matters. Yet even here the value comes from moving small messages quickly, with one writer per slot, rather than from many hosts mutating shared structures.

Realistic uses and fantasy ones

Once we put the pieces together, the uses sort themselves fairly cleanly. The deciding questions are how often more than one host writes the same lines, and what happens to shared state when a host fails.

A use-case fit matrix. Moving down the list, more hosts write the same lines and more state survives a crash.

The bottom row deserves its label, and you may well hear it proposed anyway. The CXL introduction itself cautions that a composable system is still distinct from treating all resources connected via CXL as a single big system, and that workloads will keep preferring locality and as few CXL links as possible, to limit both coherence traffic and the blast radius of failures. If someone proposes running an unmodified multi-threaded application across a rack because the memory is now coherent, they are proposing the DSM of the 1990s with a faster wire, and it will meet the same false sharing, the same contention and a worse failure story.

Where would we start, then? The middle rows are where engineering effort pays. Read-mostly sharing gets the benefit of hardware coherence with little of its cost. Message passing with clear ownership turns the low latency into faster RPCs without inviting contention. Write-shared structures can work too, but only if you design them like distributed protocols: leases instead of bare locks, fencing tokens checked by the structure itself, idempotent recovery, and a plan for a host that disappears in the middle of an update.

Distributed shared memory, re-read

Go back to the 1990s pitch. Give a cluster one address space, and the hard parts of distributed programming disappear. CXL 3.x has delivered the first half of that sentence better than any DSM system did: cache-line granularity, hardware coherence, and loads and stores at a few hundred nanoseconds instead of page faults over a network. It has not delivered the second half, and it cannot, because the hard parts were never in the memory. They were in the independence of the machines.

So the question we would ask of any design that uses CXL shared memory is not whether the memory is coherent, however tempting that question is. Ask who writes each line, how often more than one host does, and what the data structure does when a writer vanishes halfway through. If the answers are “one host”, “rarely” and “a lease expires and a fencing token protects the rest”, the design is on firm ground. If the answers are “everyone”, “constantly” and “nothing, the hardware is coherent”, we have walked back into the oldest trap in distributed systems with better hardware.

None of this makes CXL shared memory a bad idea, in our view. It makes it a distributed-systems tool with an unusually fast transport, and you should judge it the way you would judge any such tool: by its failure modes first and its latency second. Keep that order and the hardware is a genuine opportunity; reverse it, and the coherence will hide your bugs until the first host crashes in the middle of a write.

The next article turns to the question every reader of this series faces sooner or later: how to evaluate any of these claims, pooling, tiering, bandwidth or sharing, without owning CXL hardware.

References

  • Das Sharma, D., Blankenship, R., & Berger, D. S. An Introduction to the Compute Express Link (CXL) Interconnect. — fine-grained sharing as a datacenter challenge and the network delay it pays; back-invalidate channels and HDM-DB; the directory example with hosts 1, 3 and 4; directory state and sharer lists; global fabric-attached memory; independent coherency domains; read-mostly scaling and software coherence; rack-scale deployment economics; composable is not one big system

  • Chen, C., et al. Next-Gen Computing Systems with Compute Express Link: a Comprehensive Survey. — CXL sharing versus pooling and versus DSM’s network coherence; AMD’s snoop-filter caution; CXL-SHM, HydraRPC, NotNets and RPCool; the missing failure model and new failure types

  • Tanenbaum, A. S. Modern Operating Systems. — distributed shared memory through page faults; page-sized units versus cache blocks; false sharing; read-only replication for consistency; test-and-set lock ping-pong; backoff and per-waiter lock variables

  • Sun, Y., et al. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices. MICRO ‘23. — stores reaching lower bandwidth efficiency than loads on real CXL devices and remote-socket DRAM, attributed to implicit loads and cache coherence checks

  • Kleppmann, M. Designing Data-Intensive Applications. O’Reilly, 2017. — shared-memory architecture and its limited fault tolerance; partial failure; leases; fencing tokens

Engineering Deep Dives goes beyond tutorials — covering system design, performance, and the “why” behind engineering decisions. Upgrade to get every post in full.

© 2026 Shriom Tripathi
548 Market Street PMB 72296, San Francisco, CA 94104