It seems as though this is kind of a common paradigm for game engines. Because the tech and/or best practices move so quickly, trying to lock in full backwards compatibility becomes a burden and thus engine devs fall back to "If you started your game on 3.x then it's probably best to stay there. Your next game can start fresh on 4.x"

It'll take some time before you can confidently make that assessment, though. It took ten years for Perl 6 to come out, and then actually it didn't come out as Perl.

Why Pearl instead of the dozens of other successful projects that maintain a stable version while working on a new version?

Because I worked with Perl and got burned a bit so it came to mind. I understand game engines are different and people build against stable versions for years sometimes.