For sure, a small team can do the work. The question is: who will pay them to? Your five smart compiler friends could maybe get contract work making specific hotspots a bit faster for some specific medium-sized company.

But who is going to finance them rebuilding the whole approach from scratch over several years, in a way that benefits a whole platform rather than just their company, and shelter the team from short-term priorities and reorgs?

That takes a big company that has an enormous economic stake in the entire platform.

Also, Sun making Java fast was due to Hotspot, which was inspired by work on Self, the successor to Smalltalk.