mailwal

Memory Tiering in Linux: Promotion, Demotion and Cost

Engineering Deep Dives

Welcome to Engineering Deep Dives — deep, practical content for engineers who want to go beyond the surface.

Memory Tiering in Linux: Promotion, Demotion and Cost

Memory tiering is a hot/cold classifier fed stale data. How Linux demotes and promotes pages, what DAMON and TPP add, and what a wrong guess costs your tail.

Demotion, Promotion, and the Cost of Guessing Wrong

Picture a session store running on a machine with local DRAM and a CXL memory node. Overnight, traffic drops, and a few gigabytes of session objects go untouched for an hour. To the kernel they look exactly like what tiering exists to find: cold pages occupying fast memory. Reclaim demotes them to the CXL node, and although the timing will prove unlucky, the decision is correct given the evidence available.

At eight in the morning the users come back. Those pages turn hot within minutes, and every access to them now pays the CXL price. Promotion will eventually bring them home, but first it has to notice them, and it notices through a sampling mechanism that may not look at a given page again for tens of seconds. Meanwhile the pages promoted overnight, the ones that looked hot while the system idled, are now in the way. So the kernel spends the busiest part of the morning moving memory in both directions, and the latency chart shows it.

Could the kernel have known? Not from anything it had measured.

Nothing in that story is a bug. The TPP study of Meta’s production services found that cold pages often come back: for the Web service, almost 80% of pages that had gone cold were accessed again within ten minutes. Every decision in the story followed correctly from the data the kernel had. The data simply described the past, while the decision was a bet on the future.

The core insight: memory tiering is a hot/cold classifier wearing a memory-management costume, and its inputs are stale by construction. Every signal it can gather, a reference bit, a hint fault, a sampled access, describes what a page did, whereas every placement is a bet on what it will do. So the useful questions about a tiering policy are what each kind of mistake costs, and how quickly a mistake can be undone.

The previous article ended with a CPU-less node that nobody’s first touch ever fills. This one asks who decides what moves onto it, and what we pay when that decision turns out wrong.

Memory tiering is a classifier in disguise

Strip away the vocabulary of LRU lists and watermarks, and a tiering system has the shape of any classifier. Its inputs are features gathered about each page: whether a reference bit was set, where the page sits in the active or inactive LRU list, whether a sampled page raised a fault, how many sampled accesses a region collected. Its output is a label, hot or cold, and the label drives an action, keeping the page in local DRAM or moving it to the slower node. There is no training set in the usual sense. Instead, the classifier learns online, from evidence it pays to collect.

Each feature has a price, though. A reference bit is nearly free to set, because the hardware sets it, but something has to scan and clear it. A hint fault costs a trap on a page that was working perfectly well. A DAMON sample costs a check per region per interval. If you want fresher evidence, you pay more often, and if you pay less often, your evidence ages. We will meet that trade in every mechanism below, under a different name each time.

Operating systems have faced this problem since paging began, and the textbooks state its limit plainly. Tanenbaum describes the optimal replacement algorithm, which evicts the page that will not be needed for the longest time, and then notes that it is unrealizable, because at the moment of decision the operating system has no way of knowing when each page will next be referenced. Every practical algorithm since then predicts the future from the past. Tiering inherits that limit whole. It merely makes two decisions, promotion and demotion, whereas replacement made one.

The approximations also lose information in a particular way. Tanenbaum’s aging algorithm keeps a few bits of reference history per page, and he points out that by recording a single bit per interval it can no longer tell whether a page was referenced early or late within that interval. Silberschatz, Galvin and Gagne make the same point about approximating the working set with a timer and a reference bit. The result is not entirely accurate, since nothing records where in the interval a reference happened, and the remedy of more history bits and more frequent interrupts costs correspondingly more.

That trade, resolution against overhead, is tiering in miniature. We can see more if we look more often, and looking is never free.

The loop every tiering policy runs. Each dotted arrow back into the classifier is a new observation, and it already describes the past by the time it arrives.

Four outcomes, two kinds of mistake

A binary classifier has four outcomes, and naming them makes the costs visible. A hot page kept in DRAM and a cold page moved to CXL are the two correct calls. The first preserves performance, whereas the second is the entire economic point of tiering, since it frees fast memory for data that needs it.

The two mistakes differ in kind. A false cold, a hot page demoted, costs every access that follows until promotion repairs it, plus the migration that brings it back. A false hot, a cold page promoted, costs a migration and a slot of fast memory. Because fast memory is full by assumption, that slot comes from some other page, which may itself be hot, so false-hot errors tend to manufacture false-cold errors elsewhere.

We can make the asymmetry concrete with rough numbers. Take the latencies from the first article’s testbed, about 130 ns for local DRAM against 284 ns for an ASIC-based CXL device, so each access to a misplaced page costs roughly 150 ns extra. Suppose a hot page is touched 10,000 times a second in ways that miss the CPU caches, and promotion takes 30 seconds to notice it. That one false cold accumulates about 45 ms of stall, spread across whatever requests touched it, before its migration even starts. A false hot, by contrast, costs one page copy and one displaced page, which may be cheap or may be the next false cold. The cost of a false cold therefore scales with how hot the page really is, whereas the cost of a false hot stays roughly fixed per mistake.

The four outcomes. The dotted edge is why the two errors are not independent.

Ping-pong, which AutoNUMA: The Kernel’s Opinion, and When to Overrule It described as a page shuttling between nodes, is simply these two errors alternating on the same page. The asymmetry matters because tuning any classifier moves errors from one column to the other. A lower bar for promotion rescues hot pages sooner and lets more false hots through. A higher bar does the reverse. No setting removes both kinds, so the real choice is which mistake you would rather pay for, and that depends on what each one costs your workload.

Demotion: a cheaper way to be wrong

Before CXL, the cold end of the classifier fed swap. Reclaim, the machinery Virtual Memory Is a Promise the Kernel Can Break traces from overcommit to the OOM killer, picked inactive pages and wrote them to a swap device, so a false cold meant a major fault and a read from storage the next time the page was touched. That made calling a page cold expensive, and reclaim accordingly waits for memory pressure, signalled by per-zone watermarks, before it acts.

Demotion changes what that mistake costs us. The TPP authors replaced swapping with migration in the local node’s reclaim path: candidates for reclamation go onto a separate demotion list and are migrated asynchronously to the CXL node, which they note is orders of magnitude faster than swapping. Crucially, a demoted page is still in memory. If the guess was wrong, the next access is slower, yet it is still a load rather than a trip to storage. TPP chooses candidates from the inactive lists, so hot pages rarely reach the CXL node unless local memory is smaller than the hot part of the working set. If a demotion fails, TPP falls back to ordinary reclaim for that page, and the CXL node itself still reclaims to swap.

Upstream Linux offers the same trade, but as a switch rather than a redesign. The kernel’s CXL reclaim notes explain that reclaim prefers swap or zswap by default, and that setting demotion_enabled makes it prefer distant NUMA nodes whenever they have capacity, with the next demotion node chosen through the memory-tier machinery from the previous article. LWN’s summary of the tiering work puts the asymmetry neatly: demoting unused pages is a relatively easy decision that fits existing memory management, whereas deciding when to promote pages back is trickier.

Cheap demotion has a second-order effect that is easy to miss.

If a false cold costs less, a classifier can afford to call pages cold sooner, which keeps headroom in local DRAM. TPP builds on exactly that, decoupling reclamation from allocation with two watermarks. Local DRAM keeps reclaiming toward a higher demotion watermark, while new allocations need only clear a lower allocation watermark.

The headroom matters because promotion needs somewhere to land. Without the decoupling, the TPP authors found that promotion almost halted, trapped pages on the CXL node generated 55% of all memory traffic, and throughput fell by 12%. With it, promotion ran at a steady 50 KB/s on average and rose to 1.2 MB/s at the 99th percentile during surges. The authors also paired TPP with TMO and found that it effectively turned swapping into a two-stage process, demote first and swap only later, so cold pages spend a while on the CXL node before they ever reach storage. If you remember one design rule from TPP, it is probably this one: a tier you cannot promote into is a tier you cannot correct mistakes in.

If you enable demotion on a shared host, remember the caveat from the previous article as well. In kernels up to v6.15, the kernel’s notes say, demotion did not respect cpuset.mems, so a container fenced away from the CXL node could still find its cold pages moved there. A cheaper mistake is still a mistake, and it may land in a place you thought you had ruled out.

Promotion: where staleness bites

Promotion is the harder half, because it has to notice a page that nobody is watching closely. On the CXL node there is no cheap per-access signal. The kernel manufactures one with NUMA hint faults, unmapping a sample of pages so that the next touch traps, and the AutoNUMA article walks through that mechanism in detail. The original policy promoted on the first such fault. In classifier terms, that is a threshold of one observation, and it admits every page that happens to be touched once, which is the false-hot flood the TPP authors traced to ping-pong.

TPP raised the bar to two observations. When a hint fault finds a page on the inactive LRU, TPP marks it accessed and moves it to the active list instead of promoting it; only if the page is still hot at its next hint fault does it migrate. The effect on the error mix is exactly what the quadrant predicts. The promotion rate fell by a factor of 11, and the number of demoted pages that were later promoted again halved. The promotion success rate also improved by 48%, because fewer wasted promotions were filling local DRAM, and throughput rose by 2.4%. The price was paid in staleness, though: reaching peak local traffic took about five minutes longer.

Why not simply scan faster? Each scan unmaps pages that were working fine and forces a trap on their next use, so a faster scan buys fresher evidence with more faults everywhere, including on pages that will never move. That is the textbooks’ resolution-against-overhead trade again. It may explain why both TPP and the upstream kernel chose to make better use of each observation rather than to collect more of them.

Upstream NUMA balancing reached the same place by a different road. Its tiering mode is enabled by writing 2 to the numa_balancing sysctl, and at first it promoted the most recently accessed pages. Huang Ying’s 2022 hot page selection series explains why that was not good enough: even a page with a low access frequency may be touched eventually when the scan period can stretch to something like 60 seconds. The replacement measures hint-fault latency instead, the time between the scan that unmapped a page and the fault that followed. A short latency suggests frequent access, and in the patch that introduced it, pages faulting within a threshold, one second by default, count as hot.

The same patch states the cost honestly. When a cold area turns hot, its first hint fault arrives long after the scan, so the latency looks long and the pages are not promoted. Only after the next scan, perhaps a full scan period later, does the short latency appear. As a mitigation, the kernel skips the threshold and promotes everything while the fast node has free space.

That is our morning session store described from the kernel’s side. The classifier is not wrong about the past; it is waiting for enough of the present to arrive.

A page’s life under TPP-style tiering. Every edge out of the CXL states waits on a sampled observation, which is where the staleness lives.

The series also added a brake. Promotion and demotion throughput costs CPU cycles and memory bandwidth, so the kernel’s documentation for numa_balancing_promote_rate_limit_MBps warns that too much of it may hurt application latency, and suggests a limit below a tenth of the slow node’s write bandwidth (persistent memory, in the documentation’s wording). The cover letter measured the shape of the trade on a server with DRAM and persistent memory rather than CXL. Hot page selection raised the pmbench score by about 12.8% and cut the promotion rate by 25%. The rate limit then cut promotion by another 69.6%, at a cost of 1.7% of score, and automatic threshold tuning won back 4.7% while trimming promotion a further 17.1%. Fewer, better-chosen migrations beat many eager ones, which is roughly what a classifier person would predict.

DAMON: watching regions instead of pages

Hint faults give us evidence one page at a time, which is why scanning a large memory takes so long. DAMON, the kernel’s data access monitor, takes a statistical shortcut described in its design documentation. It groups adjacent pages assumed to share an access frequency into regions. In each sampling interval it checks one randomly chosen page per region and increments the region’s nr_accesses counter if that page was touched. Overhead therefore depends on the number of regions rather than the amount of memory, and you choose the trade by setting minimum and maximum region counts.

The assumption does the heavy lifting, and the documentation says so: if pages in a region do not really share an access frequency, the output loses quality. To keep the assumption true, DAMON merges neighbouring regions whose frequencies are close and splits regions after each aggregation interval. It also keeps an age per region, counting how long the region’s pattern has held, which lets a policy weigh recency against frequency.

The tuning guide then names the dilemma of every tiering classifier in a single paragraph. If the aggregation interval is too short, few accesses are captured and everything looks rarely accessed. If it is too long, regions take too long to converge for the purpose at hand. The default of 100 ms turns out to be too short in many cases, and the guide recommends a sampling interval of about a twentieth of the aggregation interval.

Notice that no setting escapes the trade. You can only pick which way to be wrong.

DAMON’s loop. Overhead scales with the number of regions; accuracy depends on the regions really being uniform.

DAMON on its own only observes, however. Its operation schemes, known as DAMOS, turn observations into actions, and the action list now includes migrate_hot and migrate_cold, which move regions while prioritising warmer or colder ones. Two controls make those schemes safer than a naive classifier. Quotas cap the time or bytes an action may consume in a period and apply it to the highest-priority regions first. Watermarks switch a scheme on only while a chosen metric, free memory for instance, sits inside a target band. Those are the rate limit and the headroom rule from the previous sections, expressed as configuration.

The age counter deserves a second look, because it measures directly something the hint-fault approach has to infer. A region that has stayed cold for many aggregation intervals is a much safer demotion candidate than one that went quiet in the last interval, yet a classifier that sees only access counts cannot tell them apart. If you run DAMON in monitoring-only mode first, you can see both numbers for your own workload before any scheme moves a single page, which is usually the cheapest experiment available.

What a wrong guess costs

Every misclassification ends in a migration, and migrations are where the classifier’s mistakes turn into latency. The AutoNUMA article covers the mechanics. The short version from the kernel’s page migration documentation is that the page’s table entries are replaced by migration entries for the duration of the move, so any thread touching the page waits on the page lock or the migration entry until the copy completes. The Demystifying CXL authors add two costs that land on the whole system: copying pages can block memory controllers from serving urgent reads for latency-sensitive applications, and updating page table entries takes a considerable number of CPU cycles and memory accesses.

What one misclassification costs, and who pays. Only the fourth cost lands on the page’s own user; the rest spread across the machine.

Their Redis experiment shows us what that adds up to. They placed all of Redis on CXL memory, let TPP migrate pages to DDR until 25% remained on CXL, and measured only after 75% of the pages had moved, to exclude the initial burst. Compared with placing 25% of the pages on CXL statically, TPP produced a 174% higher p99 latency, because it kept migrating a small fraction of pages back and forth as its hotness metric changed. The anti-ping-pong filter was in place, and the residual traffic was still enough. More generally, the authors found that for applications demanding microsecond-scale responses, the state-of-the-art placement policy raised tail latency beyond static partitioning, because of migration overhead.

That result reframes the question we should ask about latency-sensitive services. A static split never misclassifies anything, though only because it never classifies. It is wrong in a known, fixed way, and its tail is set by the fraction of accesses that happen to land on the slow node. A dynamic policy can achieve a better average, and for throughput-oriented services the TPP numbers suggest it often does: on Meta’s Cache2 workload, default Linux lost 18% of throughput while TPP lost 5%, even with 41% of traffic served from CXL.

The dynamic policy, however, trades a fixed cost for an intermittent one, and intermittent costs are precisely what a p99 measures.

Notice also when the costs arrive. Misclassifications tend to cluster around phase changes, when yesterday’s cold data turns hot or a batch job sweeps through memory it has not touched for hours. Those shifts usually coincide with load changing, so the migrations they trigger land at the moment your service has the least slack. We should expect tiering’s worst minutes to overlap with the application’s busiest ones, rather than to be spread evenly across the day.

The thrash failure mode

There is one regime where tiering gets worse the harder we let it try. Silberschatz, Galvin and Gagne define thrashing as a process spending more time paging than executing. It happens when the process cannot keep its actively used pages in memory, so every replacement evicts a page it needs again at once. Tiering has a direct analogue. When the hot part of the working set is larger than local DRAM, demotion can only choose among pages that are all in use, which is exactly the case the TPP authors exclude when they say their inactive-list selection rarely demotes hot pages.

Tiering’s version of thrashing. No labelling is correct when everything is hot, so the loop has no exit inside the classifier.

Each demotion then creates a false cold, each promotion displaces another hot page, and the machine spends its memory bandwidth moving the same pages in a circle. Nothing in the classifier can fix that, because no labelling is correct when everything is hot. The remedies, however, are the old ones: reduce the demand on the fast tier, give it more capacity, or stop moving pages and accept a fixed split.

Some research designs try to detect the condition directly. The survey by Chen and colleagues describes NeoMem, which builds on statistics from a profiler called NeoProf, adjusts its hot-page threshold in real time, and keeps a demotion flag to measure how severe thrashing has become. On a stock kernel the practical signal is simpler, if cruder: migration traffic rising in both directions while useful work falls.

How would you know you had crossed into this regime? The honest answer is that you may not, unless you already watch migration rates next to throughput. A promotion rate limit at least bounds the damage, though it does so by letting the loop run more slowly rather than by breaking it.

Choosing which mistake to pay for

So which mistake should you pay for? The answer depends on what your service is measured by, and we can be fairly concrete about it. If your service lives or dies by a microsecond-scale percentile, the Demystifying CXL numbers argue for static placement. Decide once which data may live on CXL memory, bind it there with the placement tools from Placement Policy as an API, and keep automatic promotion away from that memory. You will pay a fixed, predictable penalty on the slow fraction, and you will not pay for anybody’s guesses.

If throughput is what you are paid for, dynamic tiering is likely worth its noise, and the knobs are the ones we have met. Require more than one observation before promoting, or use the latency threshold. Rate-limit promotion well below the slow node’s write bandwidth. Keep headroom in local DRAM so that promotions can land, and enable demotion so that false colds stay cheap. Page type helps too. The TPP authors found that steering file-cache allocations to the CXL node gave their Web and Cache workloads all-local performance even with a small local node, since much of a page cache tends to be colder than anonymous memory. We would also put the promotion and demotion rates on the same dashboard as throughput from the first day, so that someone notices when the workload drifts out of the regime the knobs were tuned for, which is where tuning tends to stop paying without anyone noticing.

If you cannot tell which kind of service you run, measure before you choose. The Meta memory-subsystem study found that a small percentage of pages contributes most of the bandwidth across its production workloads, which is the precondition that makes tiering pay. Whether your service shares that skew is an empirical question, and DAMON in its monitoring-only mode is a reasonably cheap way to answer it before anything moves.

The morning, re-read

Let us return to the session store. Every step was locally correct, yet the morning still hurt: demoting pages that had been idle for an hour, waiting for evidence before promoting them, rate-limiting the migrations once the morning began.

The cost came from the gap between when the evidence was gathered and when it was used, and no tuning closes that gap. You can only decide where to absorb it.

For that service, three changes would have moved the cost somewhere tolerable. Keeping headroom in DRAM would have let the morning’s promotions land at once. A promotion rate limit would have stopped migrations from competing with requests at the worst moment. And binding the session store to local DRAM would have taken it out of the classifier altogether, at the price of capacity, which for a latency-critical store is often the cheapest price on offer.

That is the general lesson we would draw, and it is narrower than “tiering is risky”. A tiering policy is a classifier with a known blind spot in time. Before you enable one, ask what a false cold costs your workload, what a false hot costs it, and how long each takes to undo. The next article turns the problem around: when the goal is bandwidth rather than capacity, the best placement puts some hot pages on the slow tier on purpose.

References

  • Maruf, H. A., et al. TPP: Transparent Page Placement for CXL-Enabled Tiered Memory. ASPLOS ‘23. — cold-page re-access within ten minutes; limits of idle-page tracking; demotion by migration instead of swap, candidate selection and fallback; decoupled demotion and allocation watermarks and their measured effect; hint-fault sampling on CXL nodes; instant promotion and the active-LRU filter with its measured effect; Cache2 and varied-latency results; file-cache placement

  • Sun, Y., et al. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices. MICRO ‘23. — TPP versus static partitioning for Redis (174% higher p99); migration costs to memory controllers and page tables; tail latency of microsecond-scale applications

  • Wang, Y., et al. CXL-DMSim: A Full-System CXL Disaggregated Memory Simulator. — the 130 ns local and 284 ns ASIC CXL latencies used in the worked example

  • Chen, C., et al. Next-Gen Computing Systems with Compute Express Link: a Comprehensive Survey. — NeoMem’s hardware profiling, adaptive threshold and thrashing measure

  • Workload Behavior Driven Memory Subsystem Design for Hyperscale (Meta). — a small percentage of pages contributes most memory bandwidth

  • LWN, Weighted interleaving for memory tiering. — demotion as the easy decision, promotion as the tricky one

  • Tanenbaum, A. S. Modern Operating Systems. — the unrealizable optimal replacement algorithm; aging and the loss of order within an interval; the summary of replacement algorithms

  • Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed. — thrashing; the working-set window and its timer-and-reference-bit approximation and cost

Engineering Deep Dives goes beyond tutorials — covering system design, performance, and the “why” behind engineering decisions. Upgrade to get every post in full.

© 2026 Shriom Tripathi
548 Market Street PMB 72296, San Francisco, CA 94104