Now it seems clear that what’s missing is another architectural leap like transformers, likely many different ones. That could come from almost anywhere? Or what makes this something where big tech is the only potential source of innovation?
Neurosymbolic architectures are the future, but I think LLMs have a place as orchestrators and translators from natural language -> symbolic representation. I'm working on an article that lays out a pretty strong case for a lot of this based on ~30 studies, hopefully I can tighten it up and publish soon.
At best, they can sell their IP to BigTech, who will then commercialize it.
Are you saying you disagree that a new architectural leap is needed and just more compute for training is enough? Or are you saying a new architectural leap is needed and that or those new architectures will only be possible to train with insane amounts of compute?
If the latter I dont understand how you could know that about an innovation that’s not yet been made
In other words, it’s is MORE likely that an OpenAI/Google/Microsoft/Grok/Anthropic gets us closer to AGI than a startup we haven’t heard of yet. Simply because BigTech has cornered the market and has a de facto monopoly on compute itself. Even if you had raised $10 billion in VC funding, you literally can not buy GPUs because there is not enough manufacturing capacity in the world to fill your order. Thus, investors know this and capital is flowing to BigTech, rather than VC funds. Which creates the cycle of BigTech getting bigger, and squeezing out VC money for startups.
It's a bubble. The tricks keep working until they suddenly don't, and then all the prior tricks unwind themselves.
AMD did this deal because it's literally offering financing to them. OpenAI doesn't have access to capital markets like AMD does. So it's selling off shares of its own stock to finance the purchase of billions of dollars worth of GPUs. And the trick appears to be working since the stock is up 30% today, meaning it has paid for itself and then some.