How do you know massive datasets are required? Just because that’s how current LLMs operate, doesn’t mean it’s necessarily the only solution.
Then the resources needed to discover an alternative to brute-forcing a large model are a huge barrier.
I think academia and startups are currently better suited to optimize tinyml and edge ai hardware/compilers/frameworks etc.
It's going to get orders of magnitude less expensive, but for now, the capital requirements feel like a pretty deep moat.