Sure. I've been the maintainer of a perl portability module (Devel::PPPort) for a long time and every release was tested against basically every possible version (and several build flag permutations) of perl that was potentially out in the wild.
(Not meant sarcastically :-)
If you upload a module to CPAN, you automatically get it tested against a huge matrix of configurations:
http://matrix.cpantesters.org/?dist=Log-Any-Adapter-FileHand...
Very true, and it's definitely a great service!
However, the set of versions/configurations is still limited, and it can take an awful lot of time for the matrix to fill up. I've fixed a bug specific to perl-5.10.0 about a week ago and so far the module hasn't been picked up by that version again.
So while this is definitely good as a service for the general public, it doesn't get you very far if you're trying to build a thing that's supposed to ensure compatibility for other Perl modules across 20 years of Perl history. :)
@OP: Can you please explain why you keep 50 gigs of perl around? :-)
I use compressed read-only file systems all the time to save space on my travel laptop. I have one squashfs for firefox, one for the TeX base install, one for LLVM, one for qemu, one for my cross compiler collection. I suspect the gains over squashfs will be far less pronounced than for the pathological "400 perl version".