When you boost the loudness too much (for your specific turntable to handle), the needle will simply jump out of the groove, not kidding.

This is true, to a certain extent. But it doesn't mean that masters for vinyl always have more dynamics left intact.

More often than not these days, the same compressed master is used for the vinyl. To combat the groove-jumping problem, the overall level is simply dropped.

I've seen it argued that multi-disc setups are where the loudness wars started. When you can switch between CDs quickly, you start comparing how they sound. If the volume knob is left alone, the CD that is mastered louder is going to sound louder, and thus better.

Thus, people started mastering CDs for loudness.

An alternative idea of mine is simpler. Loud music is considered worse than quiet music (quiet music sounds worse, but loud music still does, and also bothers other people). So, when you need to pick a volume setting for your collection, you bias towards setting it lower, so the really loud ones don't become too loud. Thus, the quieter CDs are annoying because they always sound quiet, whilst the load CDs sound about right, because your volume is much more suitable for them.

Just guessing, but maybe the CD version is also used for radio play, and needs to be mastered as loud as possible not to lose the war?

I guess special purpose releases don't usually end up on the radio so they can be mastered for people who actually appreciate music. ;)