I used that C program as the initramfs for qemu, with it spinning up an instance based on the test kernel, and could grep for what I wanted in the output to get a relevant exit code.
Combine that git bisect's automatic bisecting process, and I was able to automatically git bisect through a few thousand commits in the kernel to find the cause. It only took it about 50 minutes to find it in the end, which was helped a bunch by ccache (and the sheer build parallelism granted by building on a 52 core system).
That was a fun, somewhat out of the blue, task.
The process I took was based on https://ldpreload.com/blog/git-bisect-run, with various adaptations to suit the needs and the relevant build environment.
is this not a slightly better kernel equivalent of 'it builds, ship it' ?
Surely, "it runs, ship it"? That seems quite a bit better.
With 16%+ lines in all 24 architectures combined, you're not going to reach more than one of them at a time.
10% is going to be an extremely happy case. If we can test 5-10% that's a great achievement. To get past that you need to start booting real hardware with specific configurations.
From triaging reports daily from kbuild test robot aka "0day" bot, I'd say 1/3 to 1/2 of kernel commits pushed by various developers to their trees have never even been compiled (very obvious mistakes regardless of toolchain).
We get way more coverage in actually shipping these kernels in Android and ChromeOS.
Also, Linaro's Tool chain Working Group runs a ton of CI on the kernel as well. There's an effort from RedHat called KCI to aggregate all of these reports.