But you've mentioned it yourself, so you presumably don't see that as a contradiction.
What do you think makes the complexity of Linux achievable with FOSS while the comparable complexity of a AAA engine not?
It just comes down to incentives. Practically every living human being stood to gain from the existence of a high quality general purpose FOSS operating system, and so Linux naturally came about.
The people who would benefit from a AAA open source game engine are... closed source AAA game development studios. As such there's really no incentive for some physics whiz to spend their extremely valuable time toiling away on something that is only going to be used for someone else's profit. The intersection of people who build game engines, and people who make games with those engines, is extremely small.
There's really no such thing as open-source game development except for hobbyists, because that would be literally giving your product away. Games are a hit based, one-and-done type product much more akin to movies than other software.
Clearly there's something because it hasn't been done. Why do you think there are no high-quality FOSS engines?
There are many more or less serious potential explanations: not enough people care, it wasn't (or still isn't) financially feasible, the people able to develop AAA engines are actively against sharing source or ideas, there's an engine development cartel actively pushing against it to keep their revenue.
Or it might just be a social or historical accident -- we may just have been extremely lucky with Linux and projects on that scale will never come again.
Or maybe everyone thought it would be a great idea but they also thought it would never work and so no one seriously ever started an open source engine. Maybe Godot will be a true AAA engine that will surpass them all in a few years.
I don't know.
In my opinion, the problem rather is that the skills that are helpful/necessary to build "typical" open source software are really different from the skills for a good game engine.
I'm not saying there aren't talented developers working in open source. But the sheer breadth of expertise required is unimaginable. You need top-of-their-field engineers from practically every conceivable discipline within CS to build a AAA game engine. Physics, lighting, rendering, networking, audio, UI, and that's just the tip of what I can think of. Engines like Rockstar's RAGE are easily on par with the Linux kernel for complexity.