Cgroup CPU Quotas on Embedded Linux: Explain Periodic Service Stalls Under Load
Correlate service stalls with cgroup CPU-bandwidth enforcement before assuming that the whole machine is overloaded. A group can exhaust its permitted CPU time while other processors are idle. Additional application threads can consume that shared allowance sooner without increasing the amount available during the period.
This guide uses upstream Linux v6.12 cgroup v2 interfaces and the fair-class bandwidth mechanism, with ordinary SCHED_OTHER work as the test scope. It does not generalize cpu.max to real-time scheduling classes. No cgroup settings, priorities or production workloads were changed; the numeric example is idealized arithmetic.
Find the real enforcement hierarchy
Identify the service process and its actual cgroup, then record every ancestor up to the relevant control boundary. Save cpu.max, cpu.max.burst, cpu.weight, the effective allowed CPUs and the service’s scheduling policy where those interfaces apply.
The v6.12 cgroup v2 reference defines cpu.max as a maximum runtime and period, both in microseconds. Its default is max 100000 on the applicable non-root interface. The word max removes that group’s bandwidth cap; it does not prove that ancestors, CPU placement or competing work impose no restriction.
Keep quota, relative weight and affinity separate. Weight influences the relative share under contention. A quota imposes a bandwidth ceiling. Affinity limits where execution can occur. Changing one does not automatically repair a problem caused by another.
Translate quota into aggregate runtime
For an illustrative cpu.max value of 50000 100000, the group receives fifty milliseconds of CPU runtime per hundred-millisecond period. This is equivalent to half of one CPU’s time on average, not a guarantee that it runs continuously at half speed.
If two threads run simultaneously on two CPUs, they can ideally consume that fifty-millisecond aggregate budget in twenty-five milliseconds of wall time. Once the relevant allowance is exhausted, remaining runnable work can be throttled until replenishment. The exact observed trace depends on scheduling, runtime slices, other constraints and burst behavior.
The bandwidth-control documentation describes runtime transferred from a shared pool to CPU-local accounting. It also documents retained local slices and burst-related caveats. Therefore the arithmetic example is not an exact prediction of every period boundary or a hard maximum service-delay formula.
Collect counters with application timing
Take timestamped snapshots of cpu.stat before and after a bounded workload interval. Compare changes in usage_usec, nr_periods, nr_throttled and throttled_usec where available. Record burst-related fields and configuration when used. Preserve the cgroup identity and counter epoch so a recreated group is not mistaken for a recovered one.
A rise in nr_throttled supports an enforcement event, but it is not a count of failed requests. throttled_usec is a controller statistic, not a direct measurement of one application’s response time. Pair it with request start/completion timestamps, runnable-state evidence and the actual workload.
Read ancestor statistics too. The bandwidth guide explains that a child can be throttled when its parent exhausts runtime even if the child still has its own allowance. Looking only at the leaf can misattribute the stall to the service’s local configuration.
Keep CPU pressure and ordinary utilization as supporting observations. Pressure can show delayed runnable work, but it does not by itself identify quota enforcement. A system-average utilization number can dilute the effect of a restricted service on a many-core machine.
Compare a burst with steady work
| Controlled case | What to compare |
|---|---|
| One admitted CPU-bound job | Runtime consumption, throttling and completed work |
| Several simultaneous jobs | Earlier aggregate budget consumption and request tails |
| Same total work spread over time | Whether burst shape explains periodic stalls |
| Busy sibling group | Ancestor bandwidth and relative-share effects |
| Same quota at different operating conditions | Useful work completed per unit of CPU time |
Use an approved, isolated test group and keep the observer outside the deliberately constrained workload. Bound the generator’s concurrency and duration, and retain a way to stop it that does not depend on the throttled service making progress.
For the initial comparison, hold input size and total requested work constant while changing only concurrency. If high concurrency worsens latency and consumes quota earlier, that supports a burst/admission hypothesis. It does not prove that reducing threads is always the best product choice; throughput requirements still matter.
Interpret periodic stalls carefully
A stall cadence resembling the quota period is a useful clue, not a unique diagnosis. Application batching, periodic I/O, timers and lock ownership can create similar patterns. Require correlated controller evidence and a controlled change that affects the predicted behavior.
CPU time is not a fixed number of instructions. The cgroup reference states that its temporal allocation does not account for execution frequency. Thermal throttling, power policy and a different CPU can reduce useful work completed within the same quota. Preserve those operating conditions in a comparison.
If the process is waiting for storage or a mutex rather than ready to execute, increasing its quota may have little effect. Follow the actual dependency before changing capacity policy. Conversely, removing a leaf limit may not help when an ancestor remains the bottleneck.
Choose a remedy with a product owner
Options include limiting admitted concurrency, reducing CPU demand, choosing a better period for the workload, adjusting a justified quota or restructuring the hierarchy. Each change affects isolation and neighboring services. Do not convert a diagnostic suggestion into an unlimited production allocation.
A shorter period with proportionally reduced quota can change stall shape while preserving the average budget, but introduces its own accounting and workload tradeoffs. Burst allowance can help some intermittent workloads but also changes interference. Select either only from measured requirements and the exact implementation limits.
Keep service readiness and overload behavior explicit. If the group cannot finish all admitted work within its budget, a bounded queue and defined rejection or degradation policy may be necessary. Letting retries multiply inside the same constrained group can make the symptom worse.
Close with correlated evidence
Deliver the hierarchy ledger, counter deltas, request timeline and controlled comparisons. State whether the cause was local quota, ancestor enforcement, ordinary contention or another wait. A repair passes when the required service behavior and isolation contract both remain valid under the approved workload.
The goal is not merely to make nr_throttled stop increasing. Some throttling may be the intended containment policy. The engineering decision is whether that policy produces acceptable application behavior, and whether the recorded evidence explains the observed stalls.
