Much of the cost is probably compute.

Very unlikely: note best GPT-4 estimates put at it $45 million

Altman said that GPT-4 training was over $100 million. And you need significant additional resources beside just the training run cost.

so like 10x less funding than OpenAI and 100x less than MSFT, AAPL, Google, Facebook, ...

I think it's fair to say when you hit the hundreds of millions of dollars mark the diminishing returns for making things happen faster have well and truly kicked in.

Perhaps the only benefit would be extra computational power yet I would struggle to understand the benefit of jumping from 500 million to 5 billion with such short timeframes.

The ability to truly not think about training run costs, throw random things on the wall to see what sticks. 10x resources is definitely a competitive advantage in LLM training.