BitBake Fetch Concurrency: Apply number_threads Without Changing Compiler Parallelism

A shared source server starts throttling a Yocto build while the machine still has ample CPU capacity. Reducing every kind of build concurrency may conceal the symptom, but it also slows unrelated compilation. BitBake provides a narrower control: the number_threads flag on a task name.

For fetching, the important unit is a concurrently running do_fetch task. It is not a count of recipes in the layer, compiler jobs, HTTP requests or network sockets. That distinction determines both where to configure the limit and what evidence can show it is working.

Put the task-type limit in global configuration

The BitBake variable-flags reference gives do_fetch[number_threads] = "2" as an example. Put this in configuration shared by the build, such as the selected local.conf or an appropriate configuration include. The value is illustrative, not a universal server capacity recommendation.

The same reference warns against setting this flag separately in individual recipes. A recipe-local value is the wrong way to give one supplier a special rate limit. It also notes that a flag value above BB_NUMBER_THREADS cannot increase concurrency beyond the overall task limit.

Check the global configuration with bitbake -e and inspect the assignment history for do_fetch and its number_threads flag. A value visible only in one recipe's environment does not establish the scheduler-wide policy you intended.

Understand what the scheduler counts

The examined runqueue implementation groups active, unfinished tasks by task name. For a name with a limit, it reads the flag from the configuration datastore and counts running instances. A candidate of that same task type is skipped when the count has reached the limit.

Consequently, two fetches from different recipes consume the same do_fetch allowance. They do not each receive two additional slots. A configure or compile task is a different type and can still be selected if its dependencies and the other scheduling conditions allow it.

This mechanism limits admission to task execution. It does not configure the subprocesses used by a fetcher or divide an aggregate network budget among them. One admitted fetch can still transfer a large repository, and a custom fetching method can have its own internal behavior.

Keep compiler parallelism independent

The PARALLEL_MAKE definition describes options passed to supported build tools during compilation. It operates inside those tasks. Changing it is not a substitute for limiting simultaneous do_fetch instances.

For example, a build can deliberately allow eight total BitBake tasks, at most two of them fetches, while retaining a separately chosen compiler job setting. Those numbers describe different ceilings. They do not promise that eight tasks will always be runnable or that the machine will use a fixed number of processes.

If compilation slows after reducing the fetch allowance, inspect dependencies before increasing the global thread count. Downstream work may simply be waiting for sources that have not yet been fetched. Spare CPU capacity cannot satisfy a source dependency.

Test the limit against the server symptom

Choose several independent recipes whose sources are not already available in the test download cache. Use a disposable build configuration and cache rather than deleting the shared download directory. Request their fetch tasks, then observe task start and completion times and count overlapping do_fetch intervals.

A warm-cache test may finish fetch tasks almost immediately and tell little about sustained server load. Conversely, counting established TCP connections is not a direct test of the task allowance. Correlate network observations with the tasks that created them.

If the task count respects the limit but throttling remains, investigate the server's actual restriction: request frequency, transferred bytes, per-account quotas or traffic from other builders. A per-build task cap cannot enforce a combined quota across independent BitBake processes. Adjust the policy at the level of the resource being constrained, while retaining a clear distinction between task concurrency and network capacity.