rtla timerlat: Separate Timer Interrupt Delay from Scheduler Wakeup Delay
Pair timerlat’s IRQ and thread observations for the same CPU and activation before deciding where wakeup delay accumulated. Both values are measured from the intended timer expiry. Adding them double-counts the early delay; subtracting unrelated maxima creates a number that may never have occurred.
This procedure uses upstream Linux and rtla sources at v6.12. It focuses on separating observation points in a timer activation, rather than classifying every source of CPU noise. No rtla command, real-time measurement workload or latency-triggered stop was run for this article. The timeline values are illustrative.
Understand the common time origin
The timerlat tracer documentation describes a timer armed for an absolute expiry. The IRQ observation records lateness when the timer interrupt path is reached. The thread observation records lateness when the awakened measurement thread runs.
Call the expected expiry t0, the IRQ observation t1 and the thread observation t2. IRQ latency is t1 minus t0. Thread latency is t2 minus t0. The difference between correctly paired values describes the additional interval between observation points, not a pure measure of scheduler implementation cost.
That additional interval can include the rest of the timer handler, other interrupts, preemption-disabled work, scheduling and competing execution before the thread runs. A large thread value therefore does not, by itself, prove that the scheduler chose the wrong task.
Pin both kernel and measurement mode
Verify CONFIG_TIMERLAT_TRACER and the actual available tracer. The v6.12 configuration selects osnoise infrastructure for timerlat. Check the installed rtla build against the target interface; a newer userspace binary or vendor kernel can change options and behavior.
The v6.12 rtla options distinguish user-space and kernel measurement threads. When neither mode is specified, the top implementation and hist implementation prefer user threads when osnoise/per_cpu/cpu0/timerlat_fd exists, otherwise falling back to kernel threads. User mode also observes transitions through that file interface. This article explicitly selects --kernel-threads for its two-point timeline.
Record the chosen mode, CPU list, period, scheduling policy, priority and test duration. Do not silently accept the documented high FIFO default on a busy target. A high-priority measurement workload can change scheduling behavior and interfere with other work; choose the experiment in a recoverable laboratory environment.
Prepare a bounded capture
| v6.12 option or setting | Decision before use |
|---|---|
| –kernel-threads | Select the mode used by the paired timeline here |
| –cpus and –period | Choose the affected CPUs and a justified sampling period |
| –duration and –priority | Bound the run and review its scheduling impact |
| –irq and –thread | Choose which cumulative latency should stop tracing |
| –trace and selected –event entries | Retain the causal detail needed after a stop |
The common options document those scope and duration controls. Confirm their local help and error returns before the proposed run. Specify a tested CPU set rather than copying a machine-specific range from an example.
Keep power and idle constraints visible. The optional cpu_dma_latency setting deliberately affects idle-exit behavior. A run with that constraint and a run without it answer different questions; do not present their difference as a kernel fix without controlling the power-policy change.
Interpret one paired activation
For a synthetic activation, suppose the IRQ value is eight microseconds and the thread value is thirty microseconds. The additional interval is twenty-two microseconds. Total thread lateness remains thirty, not thirty-eight. Pair by CPU and activation ID in the detailed trace, and retain both raw records.
If IRQ latency is already high, investigate what prevented or delayed reaching the timer’s interrupt observation. Candidate mechanisms include interrupt masking and platform or higher-priority interference, but the latency number does not identify which occurred.
If the IRQ observation is prompt but the thread observation is late, investigate the interval after the IRQ observation. Correlate the timer-handler duration, other IRQs, thread execution and scheduling evidence. A saved IRQ stack can help identify interrupted code, but it is one stack at one observation point rather than a complete history of the delay.
If both values are high, the thread inherits the earlier delay and may add more. Preserve that decomposition instead of classifying the entire thread value as a scheduling delay.
Use summaries to select, traces to explain
rtla timerlat top summarizes values by CPU; histograms summarize a distribution. They are useful for finding an affected CPU or tail, but the IRQ maximum and thread maximum may belong to different activations. Do not subtract those summary columns to estimate a worst-case post-IRQ interval.
For an activation of interest, correlate osnoise events with the same CPU and time interval. Their durations use net accounting, whereas timerlat values are cumulative or gross relative to expiry. Nested event durations and measurement overhead must be handled accordingly; summing every displayed duration can double-count execution.
Preserve buffer-loss statistics and the stop reason. If one member of a pair is missing or overwritten, the requested decomposition is incomplete. Report that limitation rather than joining it to a convenient record from a neighboring activation.
Test one hypothesis and preserve the boundary
- Capture a baseline with a stable workload and fully recorded measurement settings.
- Select a complete activation that exhibits the relevant delay.
- Identify a source-supported mechanism in the IRQ or later interval.
- Apply one reviewed correction or controlled experimental change.
- Repeat the capture and separately verify the application’s end-to-end timing.
A faster timerlat thread does not certify the application’s deadline. The application can have different priority, locking, CPU placement and downstream work. Likewise, a run without an outlier cannot establish a hard upper bound for untested workloads or platform states.
Deliver the paired records, the common-origin timeline, exact mode and settings, and the evidence behind the next source investigation. The useful outcome is a correctly localized delay interval with explicit uncertainty, not a misleading subtraction of aggregate maxima.
