b) Synthetic data sets have been shown to not be a substitute.
c) I have no idea why you are linking Moore's Law with AI. Especially when it has never applied to GPUs and we are in a situation where we have a single vendor not subject to normal competition.
While Moore's Law probably doesn't strictly apply to GPUs, it's not far off. See [1] where they find "We find that FLOP/s per dollar for ML GPUs double every 2.07 years (95% CI: 1.54 to 3.13 years) compared to 2.46 years for all GPUs." (Moore's law predicts doubling every 2 years)
https://epochai.org/blog/trends-in-gpu-price-performance#tre...
That incentive doesn’t invalidate research, but AI results are so easy to nudge in any direction that it’s hard to ignore.
So while we're not 10x-ing everything, it's not like there's no significant improvements in many places.
There have been a lot of people making these sorts of claims for years, and they nearly never end up accurately predicting what will actually happen. That's what makes observing what happens exciting.
Which is the elephant in the room.
There is no roadmap for any of these to happen and a strong possibility that we will start to see diminishing returns with the current LLM implementation and available datasets. At which point all of the hype and money will come out of the industry. Which in turn will cause a lull in research until the next big breakthrough and the cycle repeats.