Hurdle seems more software or process related than hardware no? Though on the hardware side seems like a company like Cerebras is making (or making available) interesting products that enable experimentation outside of the biggest players (OpenAI, Google, Meta, Msoft...)

Like the advent of Transformers, some smart dev could change how LLM's think. Self improvement could be built in as an optimization process. And if we don't "know" what might work, a platform could "guess" and try billions of combinations of possible improvements.

On the self improvement part they are not likely to reach super intelligence by software alone, how will they improve hardware without human+capital in the loop?