News for hackers. Stuff that matters.
Re:The "it" in AI models is the dataset by disgruntledphd2 on Thursday April 25, 2024 @04:50AM (#40155069)
This makes me sad, not because I disagree with it, but because it's basically common wisdom in the statistical and ML communities (of practitioners). In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics.

That being said, definitely if you use a linear model (like lasso) vs a tree based model (like XGBoost), you'll see differences, but once you have a flexible enough model and a lot of data, training time and inference complexity tend to become better ways to make a model choice.

Re:The "it" in AI models is the dataset by sevagh on Thursday April 25, 2024 @08:14AM (#40156547)
>In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics.

There are countless competitions, etc. on Kagi, AICrowd, or other platforms with an enforced standardized data set. Every entrant uses the same data set and there's a huge difference between the best and worst submissions.

Re:The "it" in AI models is the dataset by infecto on Thursday April 25, 2024 @08:27AM (#40156656)
> ...on Kagi,....

Did you mean https://www.kaggle.com/?

Re:The "it" in AI models is the dataset by sevagh on Thursday April 25, 2024 @08:42AM (#40156798)
Yes, thanks.
Re:The "it" in AI models is the dataset by disgruntledphd2 on Thursday April 25, 2024 @03:39PM (#40162044)
Agreed but if you look at winning submissions which i did stop doing, a lot of them do very good feature engineering which is not a model related thing.
Re:The "it" in AI models is the dataset by jstummbillig on Thursday April 25, 2024 @05:32AM (#40155313)
Why does it make you sad? It seems intuitiv and simple. And in reality of course the optimisation part is not trivial. What would we better if the "it" was more complicated?
Re:The "it" in AI models is the dataset by michaelt on Thursday April 25, 2024 @06:21AM (#40155622)
It used to be that people would get into these fields thinking ML would need specifically human insights, deep thinking, and philosophical insights about the nature of consciousness.

You would get into natural language modelling because you had a deep love of language. Because you think you're close to figuring language out in a systematic way, with just a few years more study.

There's a certain sadness, I think, in the revelation that the robots don't need the expertise of humanity's greatest experts and masters, they just need us to click all the squares that contain a motorcycle.

Re:The "it" in AI models is the dataset by sevagh on Thursday April 25, 2024 @08:16AM (#40156563)
> It used to be that people would get into these fields thinking ML would need specifically human insights, deep thinking, and philosophical insights about the nature of consciousness.

What's sadder is coming into a field pre-deciding that the way you approach it "is the right way" and can't tolerate that different mindsets can also get results.

Re:The "it" in AI models is the dataset by disgruntledphd2 on Thursday April 25, 2024 @06:30AM (#40155696)
This is 100% not why I am sad, see my other reply for information.

As an aside, it's wild how people put their own spin onto what I said.

Obviously I should have been clearer :shrug:.

Re:The "it" in AI models is the dataset by xanderlewis on Thursday April 25, 2024 @06:45AM (#40155803)
Well, you have to forgive some for making assumptions based on your choice of username…
Re:The "it" in AI models is the dataset by disgruntledphd2 on Thursday April 25, 2024 @06:58AM (#40155891)
Fair, I'm just generally disgruntled to be fair, the PhD was just the name of my soon abandoned blog.
Re:The "it" in AI models is the dataset by nicklecompte on Thursday April 25, 2024 @07:54AM (#40156345)
> in the revelation that the robots don't need the expertise of humanity's greatest experts and masters

If it's any consolation, this is more of a delusion than a revelation. Throwing gigantic piles of data at extremely primitive artificial neural networks[1] succceeds at artificial benchmarks and might get OpenAI over the line of commercial viability. But GPT still doesn't understand what numbers are, either in text or visually. Sora doesn't understand causality or object permanence. DALL-E doesn't have a clue what a dog or a cat actually is (despite the misleading claim in the post).

"Scaling is all you need" will be an ironic slogan of the upcoming AI winter. These machines have read more text and watched more videos than 1,000 humans could do in their lifetimes... and they're still dramatically dumber than mice.

[1] "primitive" relative to biological neurons, obviously they are quite fancy relative to other software.

Re:The "it" in AI models is the dataset by xanderlewis on Thursday April 25, 2024 @06:26AM (#40155661)
How do you know? We’re not there yet.
Re:The "it" in AI models is the dataset by disgruntledphd2 on Thursday April 25, 2024 @06:29AM (#40155686)
Because of the way it's presented, as if it's some vast new discovery that OpenAI have made, rather than common wisdom.

It makes me sad when people rediscover things (with massive compute in this case), that were already known.

It's very much spend a year in the lab to save an hour in the library.

Re:The "it" in AI models is the dataset by macilacilove on Thursday April 25, 2024 @06:09AM (#40155547)
Possibly because being in the business of trying to turn iq edge into money, not data edge into money.
Re:The "it" in AI models is the dataset by Xcelerate on Thursday April 25, 2024 @10:00AM (#40157691)
> the only people who think architecture/model choice makes a huge difference are n00bs and academics.

Are you referring to the current state of our best existing models or the potential future of ML? I find it incredibly hard to see how an LLM could implement the best “physically allowable” approximation to Solomonoff induction.

Then again, I thought it was extremely unlikely neural networks would have the abilities they currently exhibit, so who knows.

Re:The "it" in AI models is the dataset by eru on Thursday April 25, 2024 @10:58AM (#40158374)
We manage to train neural nets to approximate complicated data sets via rather simple process: back propagation.

It is indeed a marvel that it works nearly as well as it does.

But then again, evolution is even dumber (in the sense that it only makes random choices that thrive or perish, and can't even take gradients into account), but evolution has still managed to produce intelligent critters.

I guess when you have enough dimensions greedy approaches to optimisation / hill climbing can work well enough, even when you have challenging problems?

Especially if you are allowed to move to some meta levels. Eg evolution doesn't build planes, it built brains that can figure out how to build planes. Similarly with back propagation perhaps.

Re:The "it" in AI models is the dataset by danielbln on Thursday April 25, 2024 @10:18AM (#40157912)
This "n00b" seems to disagree with your sentiment on the importance on architecture: https://news.ycombinator.com/item?id=40155667
Re:The "it" in AI models is the dataset by disgruntledphd2 on Thursday April 25, 2024 @03:42PM (#40162078)
Unfortunately Google brain researchers have not yet discovered my brilliance, but if you read my argument it's about the data being much more important than the model. Granted transformers are a great model, but that doesn't refute my point.

Also arguments from authority are boring.

Re:The "it" in AI models is the dataset by HarHarVeryFunny on Thursday April 25, 2024 @12:24PM (#40159509)
> In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics.

The most notable voice refuting this opinion on Twitter was Yi Tay (founder of Reka.ai), who definitely does not belong to either of those categories!

Tay (ex. Google Brain) founded Reka.ai two years ago, and their latest multimodal language model is close to SOTA in performance.

https://x.com/YiTayML/status/1779895037335343521