Seccomp After a libc Update: Distinguish a Missing Syscall from a Blocked One
Identify the attempted syscall and the exact returned error before changing a seccomp policy. A library can interpret “not implemented” as permission to use an older path while treating “not permitted” as a final failure. An unchanged application can therefore behave differently after its libc changes syscall strategy.
This guide pins Linux seccomp semantics to v6.12 and the concrete clone fallback example to GNU libc 2.41. Only a pure userspace model of the return-value predicate was executed. No seccomp filter was installed or relaxed, no service policy was changed and no restricted target process was tested.
Separate three explanations for failure
A syscall may be unavailable in the kernel, rejected by a security filter before execution, or executed and rejected by its own argument or permission checks. A user-visible errno alone does not always distinguish those origins. Preserve the kernel, libc, application and effective policy identities with the first failure.
The Linux v6.12 seccomp filter reference states that SECCOMP_RET_ERRNO returns the filter’s chosen error without executing the syscall. A filter can consequently make an implemented syscall look unavailable, or make a library’s newer attempt fail with an error that does not activate its fallback.
Do not describe every ENOSYS result as proof of an old kernel. Likewise, EPERM is not uniquely a seccomp diagnosis. Correlate the syscall trace with the actual policy, available audit evidence and the same operation’s behavior in an authorized reference environment.
Read the wrapper that the application actually uses
The concrete example is sysdeps/unix/sysv/linux/clone-internal.c in the official GNU libc 2.41 archive. Where the clone3 wrapper is available, __clone_internal attempts that path and falls back to the older clone implementation when the result is minus one with errno ENOSYS. Other errors return without that fallback.
This is an internal libc implementation boundary, not a promise about every process-creation API or every libc release. The selected architecture and build assumptions affect the available wrapper and caching path. Inspect the actual caller and source revision before using this example to explain a different library.
In the non-assumed clone3 helper path, observing ENOSYS can also cache clone3 as unsupported within the process. Reusing one process while changing surrounding test conditions can therefore hide later attempts. Use a fresh program execution for each case; a fork-only child can inherit cached state through the copied process memory. Record the implementation’s caching behavior.
Write a compatibility matrix before editing policy
| Observed newer-call result | Selected glibc 2.41 internal behavior | Next question |
|---|---|---|
| Success | Return that result | Did the intended operation complete correctly? |
| Failure with ENOSYS | Use the older clone fallback | Is that fallback supported and permitted? |
| Failure with EPERM | Return failure without this fallback | Which layer produced the denial? |
| Another failure | Return failure without this fallback | Do arguments, resources or policy explain it? |
The executed fixture checked only that predicate for synthetic return/error pairs: ENOSYS failure selected fallback, EPERM failure did not, and a successful return did not become failure because of an unrelated errno value. This is arithmetic/branch evidence, not a seccomp or process-creation test.
For the product, extend the matrix with old and new libc builds, each supported kernel, the real service launcher and the active policy. Record whether the newer call was attempted, what error appeared, whether an older call followed and what the application reported.
Check architecture and policy composition
Syscall numbers are architecture and calling-convention dependent. The kernel guide warns against filtering a number without validating the architecture value. A rule copied from one ABI can match the wrong operation in another. Review native, compat and other supported calling conventions explicitly.
Also inspect the effective combined policy rather than a single source fragment. Filters can be layered, and action precedence determines the result. Adding an allow rule in one layer does not necessarily override a stronger action from another. Include container, supervisor and application-installed filters where they are part of the supported service environment.
Preserve argument restrictions as well as syscall names. A newer interface may package arguments differently, and a simple name substitution may not express the same security boundary. Do not assert equivalent enforcement until the policy mechanism can actually inspect the relevant information.
Design an isolated upgrade experiment
- Preserve the old and candidate userspace artifacts and the policy definition used to launch each service.
- Run the smallest relevant application operation in a disposable, authorized test environment.
- Capture the attempted syscall sequence and errors without recording sensitive payloads.
- Repeat through a fresh program execution, not merely a fork-only child, for each documented policy case.
- Verify both the intended operation and representative operations that must remain denied.
- Repeat the full service startup and recovery path under the final proposed policy.
The reference condition must be explicitly authorized; removing restrictions on a production service is not a diagnostic default. Use a separate test environment to isolate a policy effect. If policy modification is not approved, preserve the evidence and stop at the decision record.
Include the fallback’s own failure. Even when ENOSYS activates the older path, that path can be denied, unavailable or unsuitable for the required semantics. “Fallback happened” is an observation, not a successful compatibility result.
Choose a least-privilege repair
Possible remedies include a narrowly reviewed policy update for the new operation, a supported library configuration, or an explicitly tested compatibility response where policy designers intend the older path. None should be selected solely because it makes a startup error disappear.
Returning ENOSYS for every denied syscall can activate alternate code paths with different security or correctness properties. Returning EPERM for every unknown syscall can prevent legitimate compatibility fallbacks. Decide per operation from its semantics, implementation and permitted behavior; retain negative tests that demonstrate the original restriction still holds.
Close the upgrade with a source-linked syscall/result/fallback matrix and a result for the actual restricted service. The key evidence is that the required behavior survives the libc change while forbidden behavior remains forbidden. A successful unconfined test alone does not establish either condition.
