cgroups in Practice (Docker & K8s)

How container platforms translate user-facing flags to cgroup files

Docker Flag to cgroup File Mapping

Docker passes resource settings through its runtime to the kernel's cgroup controllers. The resulting files depend on the cgroup version, runtime, and cgroup driver; inspect the running container rather than assuming a fixed path or conversion.

CPU Mappings

--cpus 2 --> cpu.max = 200000 100000

--cpus 0.5 --> cpu.max = 50000 100000

--cpu-shares 512 --> relative CPU weight (runtime conversion on cgroup v2)

--cpuset-cpus "0,1" --> cpuset.cpus = 0,1

Memory Mappings

--memory 512m --> memory.max = 536870912 (512 * 1024 * 1024)

--memory-reservation 256m --> soft memory protection (memory.low on cgroup v2)

--memory-swap 1g --> memory.swap.max = 536870912 (1G total - 512M RAM = 512M swap)

I/O and PIDs Mappings

--pids-limit 100 --> pids.max = 100

--device-read-bps /dev/sda:10mb --> io.max = 8:0 rbps=10485760

--device-write-iops /dev/sda:1000 --> io.max = 8:0 wiops=1000

cgroup Driver: systemd vs cgroupfs

Container runtimes need to create cgroup directories and write to files. There are two approaches to managing the cgroup hierarchy:

# cgroupfs path
/sys/fs/cgroup/docker/
    <container-id>/
        cpu.max
        memory.max

# systemd path
/sys/fs/cgroup/system.slice/
    docker-<container-id>.scope/
        cpu.max
        memory.max

Kubernetes Resource Model

Kubernetes exposes requests and limits. Requests guide scheduling and CPU sharing under contention; limits constrain runtime usage. Their exact cgroup mapping depends on the resource, cgroup version, runtime, and enabled Kubernetes features.

Requests vs Limits

docker run \
  --cpus 2 \
  --memory 512m \
  my-app

resources:
  requests:
    cpu: "500m"
    memory: "256Mi"
  limits:
    cpu: "2"
    memory: "512Mi"

K8s Resource to cgroup File Mapping

QoS Classes

Kubernetes classifies Pods by their CPU and memory requests and limits. For node-pressure eviction, kubelet ranks Pods by whether usage exceeds requests, Pod Priority, and usage relative to requests. QoS is a useful shorthand for likely behavior, not the direct ranking rule.

resources: {}  # nothing specified

resources:
  requests: { cpu: "250m", memory: "128Mi" }
  limits:   { cpu: "1",    memory: "512Mi" }

resources:
  requests: { cpu: "1", memory: "512Mi" }
  limits:   { cpu: "1", memory: "512Mi" }

Pod cgroup Hierarchy

Kubernetes nests container cgroups beneath a Pod cgroup. Container resource settings apply per-container controls; the Pod cgroup is not automatically assigned a hard limit equal to their sum. Since Kubernetes 1.34, PodLevelResources is beta and enabled by default; when available, spec.resources can define an explicit shared Pod budget.

ResourceQuota & LimitRange

These are admission-time controls — they are enforced by the Kubernetes API server when pods are created, not at runtime by the kernel. They determine which cgroup values kubelet will configure.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    pods: "50"

apiVersion: v1
kind: LimitRange
metadata:
  name: container-limits
spec:
  limits:
  - type: Container
    default:
      cpu: "500m"
      memory: "256Mi"
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    max:
      cpu: "2"
      memory: "1Gi"

1. kubectl apply

   User submits pod spec

2. API Server

   LimitRange injects defaults, ResourceQuota checks totals

3. Scheduler

   Places pod based on requests

4. kubelet

   Creates cgroups with the computed values

HPA and VPA

Resource metrics originate from node and container accounting, then reach autoscalers through Kubernetes metrics APIs. The HPA acts on those API metrics and the requests declared for the targeted pods; it does not calculate utilization from cpu.weight.

The Metrics Pipeline

1. cgroup Files

   cpu.stat, memory.current, etc.

2. kubelet

   Reads cgroup files, exposes /stats/summary

3. metrics-server

   Scrapes kubelet, aggregates into metrics API

4. HPA / VPA

   Reads metrics, adjusts replicas or resources

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
spec:
  targetRef:
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Recreate"
  resourcePolicy:
    containerPolicies:
    - minAllowed:
        cpu: "100m"
      maxAllowed:
        cpu: "4"

The Full Picture: From User Config to Kernel Enforcement

Here is the complete chain from a Kubernetes manifest to kernel-level enforcement:

1. User writes a pod spec with resources.requests and resources.limits
2. API server admission: LimitRange injects defaults if missing. ResourceQuota checks namespace totals. Pod is rejected or accepted.
3. Scheduler places the pod on a node with enough allocatable resources (based on requests, not limits).
4. kubelet on the target node asks the container runtime (containerd/CRI-O) to create the container.
5. kubelet and the runtime configure cgroups (via systemd or direct cgroupfs), including applicable cpu.max, cpu.weight, memory.max, and pids.max values. On cgroup v2, memory protection from requests requires a configured Memory QoS reservation policy; it is absent by default.
6. Linux kernel applies the configured CPU, memory, and I/O controls during execution.
7. kubelet reads cgroup accounting files (cpu.stat, memory.current) and exposes them via /stats/summary.
8. metrics-server exposes resource metrics to the Kubernetes metrics API. HPA can use these metrics to change replica count; an installed VPA can recommend or apply new resource requests.

References

• Kubernetes: Pod QoS and Memory QoS
• Kubernetes: requests, limits, and Pod-level resources
• Kubernetes: node-pressure eviction order
• Kubernetes: HPA resource utilization
• Linux kernel: cgroup v2 controls