Also, all that needs to be set up. If you need to change some little thing, you're much faster doing it locally, rather than trying to wrap your head around the build server configuration and figure out how to just run this one test different but only for this branch and not for the normal builds ...
It happened, when the 2nd gen TR arrived. It used the same mainboards, so all the manufacturers issued BIOS updates.
Unfortunately, these updates claimed to support SEV (Secure Encrypted Virtualization). Linux of course tried to initialize it at boot/module load time and the entire thing went hanging, because TR CPUs do not support SEV, only EPYCs do.
So there were the following fixes:
1) downgrade BIOS back to pre-TR2 version,
2) blacklist the ccp module; which would make kvm_amd non-functional,
3) wait for a fix in Linux kernel, which initializes SEV with a timeout.
So it wasn't that tragic issue, if you had first gen TR.
When it was bad, it was so bad. I never wanted to know as much as I do now about IOMMU groups and PCIe lanes.
As an example, I worked a bunch on our PostgreSQL infrastructure, but we support 5 years of versions. So you have to CI/integrate your changes and test them across all 5 versions, every time you test. My machine could do this on the order of 2-3 minutes -- recompiling everything from scratch every time, and running all integration tests on all versions. There's no CI system or cost-effective cloud hardware that would have given me turnaround like that. In the end I was doing hundreds of builds over the course of just a few days.
In contrast, at $WORK we have a C codebase that takes < 1 minute to compile on my 4 core work laptop. YMMV.