I've been super happy with Gentoo in production for at least a decade. I don't see the issue? I have Gentoo on my desktop+laptop too. But, I also run loads of Debian server systems too. My worst upgrade experience has been Ubuntu systems

Does that mean, you compile updates elsewhere and load the binaries on production?

(Many) years ago I knew someone who was very happy running their business off Gentoo. "It took two weeks to "make world" with the ultimate optimization settings. "But it runs great!"

I never could bring myself to ask if they were ok with being down for 2 weeks if they needed to re-install. Say, in a case where they had an intrusion and didn't trust their backups...

There's no reason they'd have to be down for two weeks in that case. They could use a binary distribution to immediately get back into shape, and then switch over after compilation's finished, or they could compile without -OX and it'd take almost nothing, and then they'd be good to go.

Really, though, nowadays compiling everything, even compiling things for optimal performance, doesn't take that long. Your production machines shouldn't have web browsers, and without the C++ garbage behemoths on it, compilation takes like half an hour on most machines.

I'm not a Gentoo user, but it's kind of silly to act like it makes recovering substantially harder than any other distribution.

This was at a time when compiling just the kernel took 30 minutes, so YMMV. But obviously, I don't know. Just didn't seem to me like the choice is want to make for running a business. Again, ymmv...

I imagine anyone doing this seriously would have a spearate system for building binaries. Obviously if that system was compromised, you would not be in a great situation, but it should be much easire to lock that down.

Do you find debian testing to be stable enough compared to debian stable, for servers?

For similar reasons (stability), I always go for the current stable (although I've a couple machines on oldoldstable, because it takes a while to upgrade...).

fwiw, i run debian unstable in production for 6+ years, updating every few weeks, with 0 issues.

i would assume that testing should be rock solid.

I run Arch Linux for work even though we're a Debian shop, and it has a great many efficiency advantages. I think in hours they are more than worth the three hours of productivity loss it would take me to repair my machine in case something bad happens. Indeed, AUR and the extreme adaptability of Arch probably saves me 2-3 hours a week.