Understanding Cgroups

Cgroups

In the previous article we saw how namespaces make one machine look like several: each group of processes gets its own hostname, its own pid 1, its own network. But we ended on a problem. A process with all of that can still eat every CPU core and every byte of RAM on the box. Namespaces decide what a process can see. They say nothing about how much it can use.

That’s what cgroups (short for control groups) are for. A cgroup is a group of processes that the kernel keeps track of together, so you can say things like “this group gets at most 2 GB of memory” or “this group can’t have more than 20 processes”. If you use containers, you’re already using them.

We’ll start with the basic ideas, then build a cgroup by hand and watch it stop a process from forking. After that we’ll open up the kernel and see how it works, one piece at a time.

📌 A note on scope

Everything here is focused on Linux 7.3 and cgroup v2. I’m leaving cgroup v1 out entirely. As always, this is a deliberate simplification.

Let’s begin by going over the basic concepts before we touch anything.

The Basic Ideas

Here’s a small machine running a web app, with its processes split into cgroups.

Each box is a cgroup, which is just a set of processes. Cgroups can live inside other cgroups, so together they form a tree. The tree always starts with one root cgroup, and in our machine it contains web, which in turn contains proxy and app. Every process on the machine belongs to exactly one of these cgroups. Take app, for example: api-server and database live in the same group, so the kernel treats them as one unit. Now look at proxy. When nginx forked its worker, the worker ended up in proxy too. That’s the default behavior: a new process starts in its parent’s cgroup, so a whole family of processes stays together without anyone having to move them.

Now, let’s talk about limits (the text in orange). web can use at most 4 GB of memory, and that 4 GB covers everything inside it: nginx, its worker, api-server and database all share it. app sets its own, stricter limit of 1 GB, which only applies to the processes inside app. proxy sets no limit at all, but it still can’t grow past web’s 4 GB, and it has to share that 4 GB with app. A group can be stricter than the one it lives in, but a looser limit simply has no effect.

There’s one more rule hiding in there: web has no processes of its own. A cgroup can contain other cgroups or processes, but not both, so processes always live in the leaves of the tree. The reason is simple: if a process lived directly in web, it would compete with proxy and app for the same 4 GB, and there’s no good answer to who should get how much. This is the no-internal-process rule. The root is the one exception, which is why systemd can sit directly in root.

The groups themselves don’t limit anything, though. They’re just bookkeeping: the kernel knows which processes belong to which group. That memory.max = 4GB has to come from somewhere, so here’s who’s actually behind those limits.

Those are the controllers. A controller is the part of the kernel in charge of one resource: cpu hands out CPU time, memory caps RAM, io shares disk bandwidth, pids limits how many processes a group can have, and cpuset pins a group to certain CPUs. The memory.max = 4GB on web is just us telling the memory controller that web can’t use more than 4 GB. From then on, the controller counts how much memory web uses and steps in when it goes over.

So the work splits in two. The cgroup core keeps the tree: it creates and removes groups, moves processes between them, and handles a few things that act on a whole group at once, like freezing it, killing it, or telling you when it becomes empty. The controllers plug into that tree and do the actual resource accounting and limiting.

Now let’s see what all of that looks like from the shell.

Interacting with Cgroups

One of the interesting things about cgroups is how you interact with them: there’s no special syscall for them. Instead, the kernel exposes them as a filesystem mounted at /sys/fs/cgroup, and you manage them with plain mkdir, echo and cat. Let’s build one step by step (you’ll need a root shell).

First, let’s look at the root of the tree:

# cd /sys/fs/cgroup
# cat cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc dmem

That list is the set of controllers the root has switched on for the groups below it. Every controller we just talked about is there, plus a few more specialized ones.

Now let’s create our own cgroup. Yes, it’s just a directory:

# mkdir demo
# ls demo
cgroup.controllers      cpu.max        io.stat          misc.max
cgroup.events           cpu.stat       memory.current   pids.current
cgroup.freeze           cpu.weight     memory.max       pids.max
cgroup.kill             cpuset.cpus    memory.stat      pids.peak
cgroup.procs            cpuset.mems    memory.swap.max  rdma.max
cgroup.subtree_control  io.max         misc.current     ...
...

That’s all it takes. demo is a new cgroup, and the kernel has already filled it with files (I’ve trimmed the list, there are more than 80 of them). The cgroup.* files are the core ones that every cgroup gets. All the rest come in families, one per controller: cpu.*, memory.*, io.*, pids.* and so on. demo has files for every controller because the root switched all of them on for its children, which is exactly what that cgroup.subtree_control list was telling us.

Now let’s create a subgroup inside demo, called build, and limit the number of processes it can have. For that we need the pids controller. Controllers are switched on by the parent, for its children, so first we ask demo to enable pids for whatever we create inside it. Then we create build:

# echo +pids > demo/cgroup.subtree_control
# mkdir demo/build
# ls demo/build
cgroup.controllers      cgroup.stat             io.pressure
cgroup.events           cgroup.stat.local       memory.pressure
cgroup.freeze           cgroup.subtree_control  pids.current
cgroup.kill             cgroup.threads          pids.events
cgroup.max.depth        cgroup.type             pids.events.local
cgroup.max.descendants  cpu.pressure            pids.max
cgroup.procs            cpu.stat                pids.peak
cgroup.pressure         cpu.stat.local

This listing is much shorter. build still gets all the core cgroup.* files, plus a few statistics files the kernel keeps for every group (cpu.stat and the *.pressure ones). But the only controller files are the pids.* ones, because pids is the only controller demo switched on for its children. No memory.max, no cpu.max: those limits simply don’t exist for build.

The one we want is pids.max. Let’s cap the group at 20 processes (strictly speaking, the pids controller counts tasks, so threads count too, but our example only uses single-threaded processes):

# echo 20 > demo/build/pids.max

The last step is to move our shell into build. We do that by writing its pid ($$) into cgroup.procs, and then we can ask the kernel where we ended up:

# echo $$ > demo/build/cgroup.procs
# cat /proc/self/cgroup
0::/demo/build

Our shell now lives in build, and anything it starts from now on lands there too.

Now let’s try to start more than 20 processes:

# for i in $(seq 30); do sleep 600 & done
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
...

Around the twentieth sleep, fork() starts failing. Nothing got killed. The kernel just refused to create any more processes.

So our limit works. Before we see how it actually stopped fork(), let’s look at what the kernel keeps in memory to make all of this happen.

How It Looks in Memory

So far we’ve only seen cgroups from the outside, as directories and files. Let’s take our example and see what the kernel builds behind it. I’ve drawn our shell and two of its sleep processes, and the controller state they lead to in build and demo.

Start with the tasks. If you read the namespaces article , this will look familiar: there, every task_struct (the kernel’s struct for a process or thread) had an nsproxy pointer to its set of namespaces. Cgroups use the same trick. Every task has a cgroups pointer that says which cgroup it’s in, but here’s the first surprise: it doesn’t point to a cgroup directly. All three of our tasks point to the same css_set instead.

A css_set is a small table with one slot per controller. Each slot points to a css (short for cgroup subsystem state), which is where a controller keeps its data for one cgroup. Each controller needs to track different things, so each one has its own struct with a css inside: pids_cgroup for pids, mem_cgroup for memory, and task_group for cpu. For example, the orange arrow goes from the pids slot to build’s pids_cgroup. There we find how many processes build has right now (counter) and the limit we set earlier (limit = 20).

The last field is dfl_cgrp. Notice that the slots don’t all point to the same cgroup: pids points to build, but memory and cpu point to demo. So the slots alone can’t tell us where our tasks actually live. That’s what dfl_cgrp is for: it points to the cgroup the tasks are in, which for our shell is build.

So the hot path, from a process to its limits, is just two pointer jumps: task to css_set, css_set to css. Notice that the cgroup itself isn’t even on that path. Let’s zoom in, starting with the controllers and their csses.

Controllers and Their State

Every limit we set belongs to a controller, so let’s start by looking at what a controller is inside the kernel. Here are the three we’ve been using: pids, memory and cpu.

Each card is a struct cgroup_subsys (include/linux/cgroup-defs.h:784), the kernel’s description of one controller. The pids one, for example, is pids_cgrp_subsys (kernel/cgroup/pids.c:454). If you remember the ops table from namespaces, it’s the same idea: a list of functions the kernel calls at the right moments. There are quite a few of them, so let’s just look at the ones that matter for us.

The first pair, css_alloc and css_online, is how a controller creates its css. Whenever a cgroup needs one, like when we switched on pids for build, the kernel calls them to create the css and get it ready. css_offline and css_free are the other side of the story: they clean everything up when that state goes away.

Then there’s can_attach and attach, which the kernel calls when a task moves into a cgroup. Notice the split: can_attach gets the chance to say no before anything happens, and only if every controller agrees does attach do the actual work.

Finally, can_fork, fork and exit are called around the creation and exit of a task. From this group, the pids controller only uses can_fork: that’s where it adds a new process to its count, and exactly where our forks got refused. To take a process off the count, it uses a different callback, release, which runs once an exited task is finally cleaned up.

The pids controller is quite simple: its css only holds a counter and a limit. So let’s look at two more challenging controllers, memory and cpu, starting with memory.

The Memory Controller

Here’s what the memory controller keeps in demo’s css, with the files you use to read and set each field.

The memory controller wraps its css in a struct mem_cgroup (include/linux/memcontrol.h:202), and the most important part of it is memory, a struct page_counter (include/linux/page_counter.h:10). It’s just a counter with some limits attached. usage is how much memory the group is using right now (what you see in memory.current), and max, high, min and low are the limits you set through the files with the same names. There’s a second counter, swap, that does the same for swap, plus some statistics.

When a process in the group asks for more memory, the charge goes up the tree through the parent pointer: page_counter_try_charge() (mm/page_counter.c:118) adds it to demo’s counter, then to the root’s, checking each one against its max. If it doesn’t fit, the pids controller would just say no, but the memory controller first tries to reclaim memory from the group, for example by dropping file caches. Only if that isn’t enough does it call the OOM killer to kill a process, and only inside that group (try_charge_memcg(), mm/memcontrol.c:2645). That’s why a container that hits its memory limit loses one of its own processes while the rest of the machine doesn’t notice.

Memory is about how much a group can hold at once. CPU is about time, so its controller works in a very different way.

The CPU Controller

Here’s what the cpu controller keeps in demo’s css, which files set which fields, and a timeline of what its hard cap does over time.

This time the css lives inside a struct task_group (kernel/sched/sched.h:478), and it gives you two very different knobs.

The first one is shares, which you set through cpu.weight. This one isn’t really a limit, it’s about being fair. When the CPUs are busy, a group with weight 200 gets twice as much CPU time as a group with weight 100. But if nobody else wants the CPU, a group can use all of it.

The second knob is a real limit, cfs_bandwidth (kernel/sched/sched.h:451), which you set through cpu.max. You write two numbers, a quota and a period, like 50000 100000, which means “at most 50 ms of CPU time every 100 ms”. The timeline shows how that plays out. At the start of each period, the group gets a fresh runtime equal to its quota. Its tasks spend it as they run, and when it reaches 0 the group is throttled: none of its tasks can run user-space code until the next period starts and the runtime is refilled (throttle_cfs_rq(), kernel/sched/fair.c:7210). A task in the middle of a syscall gets to finish it first, and stops when it returns to user space. And that’s all docker run --cpus=0.5 really does under the hood.

With this, we have a good idea of how controllers work, so let’s put all the pieces together and replay our example from the kernel’s side.

Our Example, Step by Step

Here’s everything that happened in our example, from creating build to the fork that got refused. For each step you can see the command we ran, the kernel functions it triggered, and how build’s pids counter and limit changed.

1. Creating build. When we run mkdir demo/build, the kernel doesn’t create a normal directory. It calls cgroup_mkdir() (kernel/cgroup/cgroup.c:6116), which creates the new cgroup and gives it the usual cgroup.* files. Then it checks which controllers demo has switched on for its children. There’s only pids, so it creates a pids css for build (css_create(), kernel/cgroup/cgroup.c:5914). To do that it uses the css_alloc callback we saw earlier, which for pids is pids_css_alloc() (kernel/cgroup/pids.c:78). The new css starts with the counter at 0 and no limit (“max”). Memory and cpu weren’t switched on, so build doesn’t get a css for them.

2. Setting the limit. This is the simplest step. When we write 20 into pids.max, the kernel calls pids_max_write() (kernel/cgroup/pids.c:306), which just stores that 20 in build’s limit. It doesn’t check anything else. The limit isn’t used until someone tries to fork.

3. Moving the shell into build. When we write the shell’s pid into cgroup.procs, the kernel calls cgroup_attach_task() (kernel/cgroup/cgroup.c:3090). Since a task doesn’t point to a cgroup but to a css_set, the first thing it needs is the right css_set for “a task in build” (find_css_set(), kernel/cgroup/cgroup.c:1207). For pids, it uses build’s css. For memory and cpu, build has no css, so the kernel goes up to the parent and uses demo’s (cgroup_e_css_by_mask(), kernel/cgroup/cgroup.c:514). This is what’s called the effective css. If a css_set with those same csses already exists, the kernel reuses it. If not, it creates a new one.

Then the kernel tells each controller that the task is about to move. The pids controller handles it in pids_can_attach() (kernel/cgroup/pids.c:200): it adds 1 to build’s counter and takes 1 away from the group the shell was in before. Here’s a curious detail: the limit isn’t checked here. You can move a process into a group that’s already full, and the counter simply goes over. The only thing the limit stops is forks. When it’s done, the shell lives in build, and build’s counter is 1.

4. A fork that fits. Now the loop starts the first sleep. Before creating the new process, fork() asks the controllers whether it’s allowed (cgroup_can_fork(), kernel/fork.c:2426). For pids, that ends up in pids_try_charge() (kernel/cgroup/pids.c:166), which does something very simple: it adds 1 to build’s counter and checks that it doesn’t go over its limit, and then does the same for demo. build’s counter goes from 1 to 2, which is below 20, so all good. The fork goes on, and at the end the sleep gets the same css_set as the shell (cgroup_post_fork(), kernel/cgroup/cgroup.c:7057), so it lives in build too.

5. A fork that doesn’t fit. Every sleep repeats step 4, and the counter keeps going up until it reaches 20: the shell plus 19 sleeps. On the next fork, pids_try_charge() adds 1 and gets 21, which is more than 20. So it undoes the addition and returns the error -EAGAIN. The controller also records the failure (pids_event(), kernel/cgroup/pids.c:243): it adds 1 to the max line of build’s pids.events file, and the first time it also prints “cgroup: fork rejected by pids controller in /demo/build” to the kernel log. The half-created process is thrown away, fork() fails, and bash shows “Resource temporarily unavailable” and tries again.

Making room again. The counter also goes down. When a sleep finishes and the kernel cleans it up, it calls pids_release() (kernel/cgroup/pids.c:299), which takes 1 away from build and demo. That frees a slot, and the next fork fits again.

Since the addition happens at every level of the tree, a limit on demo would also affect build. If demo had a limit of 10, our loop would stop at 10 processes, even though build allows 20. A group can never get around its parent’s limit.

So that’s how a process gets counted, from creating its cgroup to the fork that gets refused. Let’s step back and keep just the ideas worth remembering.

Summary

As we’ve seen, a cgroup is just a directory. You create it with mkdir, and you configure it by reading and writing files, but each of those files is really a kernel function in disguise. The groups themselves are only bookkeeping: they form a tree, and every process lives in exactly one of them. The actual limiting is done by the controllers, like pids, memory and cpu, each one in charge of one resource.

Inside the kernel, every controller keeps its data for each group in a css, and every task points to a shared css_set that says which css to use for each controller. If a controller isn’t switched on in your group, you get the css of the nearest parent that has it, the effective css. So getting from a process to its limits takes just two pointer jumps.

Each controller then does its own thing with that data. The pids controller just counts and says no when the count goes over the limit. The memory controller tries to reclaim memory before it calls the OOM killer. And the cpu controller splits time fairly with cpu.weight and caps it with cpu.max. But they all share one rule: charges go up the tree, so a group can never escape its parent’s limits.

And remember the split from the last article: namespaces change what a process can see, and cgroups limit how much it can use. A container is mostly just a process with both.

In the next article, we’ll look at eBPF. With namespaces and cgroups, we were only turning knobs the kernel already had. eBPF goes one step further: it lets you hand the kernel a small program of your own and have it run safely inside the kernel, every time something you care about happens.

This email was sent as part of the Internals for Interns newsletter.
Visit the blog • Read this post online

You're receiving this because you subscribed to our newsletter.