There are countless competitions, etc. on Kagi, AICrowd, or other platforms with an enforced standardized data set. Every entrant uses the same data set and there's a huge difference between the best and worst submissions.
Did you mean https://www.kaggle.com/?
You would get into natural language modelling because you had a deep love of language. Because you think you're close to figuring language out in a systematic way, with just a few years more study.
There's a certain sadness, I think, in the revelation that the robots don't need the expertise of humanity's greatest experts and masters, they just need us to click all the squares that contain a motorcycle.
What's sadder is coming into a field pre-deciding that the way you approach it "is the right way" and can't tolerate that different mindsets can also get results.
As an aside, it's wild how people put their own spin onto what I said.
Obviously I should have been clearer :shrug:.
If it's any consolation, this is more of a delusion than a revelation. Throwing gigantic piles of data at extremely primitive artificial neural networks[1] succceeds at artificial benchmarks and might get OpenAI over the line of commercial viability. But GPT still doesn't understand what numbers are, either in text or visually. Sora doesn't understand causality or object permanence. DALL-E doesn't have a clue what a dog or a cat actually is (despite the misleading claim in the post).
"Scaling is all you need" will be an ironic slogan of the upcoming AI winter. These machines have read more text and watched more videos than 1,000 humans could do in their lifetimes... and they're still dramatically dumber than mice.
[1] "primitive" relative to biological neurons, obviously they are quite fancy relative to other software.
It makes me sad when people rediscover things (with massive compute in this case), that were already known.
It's very much spend a year in the lab to save an hour in the library.
Are you referring to the current state of our best existing models or the potential future of ML? I find it incredibly hard to see how an LLM could implement the best “physically allowable” approximation to Solomonoff induction.
Then again, I thought it was extremely unlikely neural networks would have the abilities they currently exhibit, so who knows.
It is indeed a marvel that it works nearly as well as it does.
But then again, evolution is even dumber (in the sense that it only makes random choices that thrive or perish, and can't even take gradients into account), but evolution has still managed to produce intelligent critters.
I guess when you have enough dimensions greedy approaches to optimisation / hill climbing can work well enough, even when you have challenging problems?
Especially if you are allowed to move to some meta levels. Eg evolution doesn't build planes, it built brains that can figure out how to build planes. Similarly with back propagation perhaps.
Also arguments from authority are boring.
The most notable voice refuting this opinion on Twitter was Yi Tay (founder of Reka.ai), who definitely does not belong to either of those categories!
Tay (ex. Google Brain) founded Reka.ai two years ago, and their latest multimodal language model is close to SOTA in performance.
That being said, definitely if you use a linear model (like lasso) vs a tree based model (like XGBoost), you'll see differences, but once you have a flexible enough model and a lot of data, training time and inference complexity tend to become better ways to make a model choice.