For sure, a small team can do the work. The question is: who will pay them to? Your five smart compiler friends could maybe get contract work making specific hotspots a bit faster for some specific medium-sized company.

But who is going to finance them rebuilding the whole approach from scratch over several years, in a way that benefits a whole platform rather than just their company, and shelter the team from short-term priorities and reorgs?

That takes a big company that has an enormous economic stake in the entire platform.