I have and it's not really that different from what works on Windows: Compile against the base system you want to support and then ship your own libraries for all other dependencies, statically linking those that you can.
Could there be things to make this more convenient? Sure - one thing I'd love to see is for glibc to support a WINVER analogue to target older versions with the same installed headers. But that is only for convenience - you can just build on older (LTS) distros or assemble your own toolchain if you need newer compiler versions (not as hard as it sounds, there are plenty of resources out there).
> fragment that minority further into Arch users, Debian users, CentOS users, OpenSUSE users, Ubuntu users
How does this matter to binary software distributions? What Distro-specific issues have you encountered that weren't just Arch-has-a-newer-library-so-I-get-an-early-warning-for-compatibility-issues-that-Ubuntu-will-have-in-the-next version.
> all with different glibc versions
So? Compile against the oldest glibc you want to support. All newer ones are backwards compatible and will have no problem running your program.
> Modern tools like AppImage make this at least somewhat standard, but Windows has had sane, standardised installers for almost 3 decades now.
Linux theoretically also has a standard package format (RPM according to the LSB). But just like with MSI on Windows, people tend to roll their own thing for various reasons - and it really is not a problem at all.
For me, I compiled a binary on my Windows machine, manually put it into an MSI alongside the the required VC++ redistributable, and it worked out of the box on every single system from Vista through to current version Windows 10 (and probably 11 too) - x86 and x64, with no issues whatsoever. If I had used the XP-compatible version of VS2015 I could have targeted that too.
Similar story for Mac (although distribution there has since become a minor pain in the arse with code signing and notarisation).
For Linux, I built a .deb manually on Ubuntu 16.04, specified the dependencies just like with Windows and uploaded it. Obviously this crashed violently on most things that weren't modern (for the time) and Ubuntu-based.
In the end, I fixed it exactly as you suggested - I built on an older LTS distro (on a CI server by this stage), shipped all of my own libraries and aggressively made sure that it would load those versions and only those versions. This still required some random hack to work on certain modern systems, as it needed a hack to support both modern glibc and a compatibility version bundled with the app.
These days, I could get a reliable Linux release up in a heartbeat, but this is only after learning the details of a fickle system through multiple iterations of user interaction.
Linux accounts for about 5% of my users, and for whatever reason the community thought it would be a good idea to fragment that minority further into Arch users, Debian users, CentOS users, OpenSUSE users, Ubuntu users (14, 16, 18 and 20, all with different glibc versions).
Modern tools like AppImage make this at least somewhat standard, but Windows has had sane, standardised installers for almost 3 decades now.