News for hackers. Stuff that matters.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by satvikpendem on Thursday December 22, 2022 @03:32PM (#34098255)
I just submitted an article about a paper by Deepmind whose main conclusion is that "data, not size, is the currently active constraint on language modeling performance" [0]. This means that even if we have bigger models, with billions and trillions of parameters, they are unlikely to be better than our current ones, because our amount of data is the bottleneck.

TFA though also reminds me of the phenomenon in mathematical proofs where some long winded proof eventually comes up but then over time it becomes simplified as more mathematicians try to optimize it, in much the same way as programmers with technical debt ("make it work, make it right, make it fast"), such as with the four color theorem that was until now computer assisted but it seems there is a non computer assisted proof out [1].

I wonder if the problem in TFA could itself be solved by machine learning, where models would create, train, and test other models, changing them along the way, similar to genetic programming but with "artificial selection" and not "natural selection" so to speak.

[0] https://news.ycombinator.com/item?id=34098087

[1] https://news.ycombinator.com/item?id=34082022

Re:Do Simpler Machine Learning Models Exist and How Can We F... by hooande on Thursday December 22, 2022 @04:17PM (#34098817)
two thoughts

People love to say "it's early" and "it will improve" about ChatGPT. but amount of training data IS the dominant factor in determining the quality of the output, usually in logarithmic terms. it's already trained on the entire internet. it's hard to see how they'll be able to significantly increase that.

And having models build models is drastically overrated. again, the accuracy/quality improvements are largely driven by the scale and diversity of the dataset. that's like 90% of the solution to any ml problem. choosing the right model and parameters is often a minor relative improvement

Re:Do Simpler Machine Learning Models Exist and How Can We F... by tlarkworthy on Thursday December 22, 2022 @04:46PM (#34099218)
All the books. I think books might be better
Re:Do Simpler Machine Learning Models Exist and How Can We F... by recuter on Thursday December 22, 2022 @04:58PM (#34099372)
There's only about 100m books (in English). About the same volume of text as the web all total.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by MonkeyClub on Thursday December 22, 2022 @05:56PM (#34100050)
Generally a book has deeper thinking than a webpage, though, I think that’s the crux of the GP’s clarification.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by disgruntledphd2 on Friday December 23, 2022 @05:29AM (#34104237)
And the distribution of text will be very different for all books vs the internet.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by albert_e on Thursday December 22, 2022 @11:57PM (#34102468)
Yes.

And (with scary privacy implications) maybe the next frontier is capturing all spoken language uttered by people in real-time and streaming it into a model that is being updated in near-real-time.

And finally brain implants extracting unspoken thoughts and neural activity and combining it all. Extend it to non human life forms as well. (universal consciousness?)

Re:Do Simpler Machine Learning Models Exist and How Can We F... by akomtu on Friday December 23, 2022 @03:06AM (#34103393)
It's the future where everyone is a universal captcha solver for the needs of the Great AI. No freedom even in your thoughts.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by theGnuMe on Thursday December 22, 2022 @04:46PM (#34099223)
> it's hard to see how they'll be able to significantly increase that.

With feedback.

Re:Do Simpler Machine Learning Models Exist and How Can We F... by DenisM on Thursday December 22, 2022 @10:18PM (#34102001)
Yes. Which is why ChatGPT is open to public.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by not2b on Thursday December 22, 2022 @05:13PM (#34099556)
Perhaps some sort of adversarial network approach could work better; models that learn to generate text and other models that try to distinguish AIs from humans, competing against each other. Also, children learning language benefit from constant feedback from people who have their best interest at heart ... that last part is important because of episodes like Microsoft's Tay where 4chan folks thought it would be fun to turn the chatbot into a fascist.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by ronsor on Thursday December 22, 2022 @09:00PM (#34101511)
I think OpenAI has already published some research showing humans preferred smaller/fewer parameter models that were "better trained" through the use of human feedback.

If there were a model that could adequately replace the role of the human, then that approach would probably work well.

Re:Do Simpler Machine Learning Models Exist and How Can We F... by charcircuit on Thursday December 22, 2022 @11:20PM (#34102308)
Passing the Turing test isn't the goal. You can have a useful model that isn't human like and can have a useless model that you can't tell isn't a human.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by not2b on Friday December 23, 2022 @03:16PM (#34110138)
Certainly. Approaches that have done better on the Turing test have used various tricks to avoid their lack of understanding, like playing a paranoid person. But some of the best chatbots give themselves away by getting stuck in a loop or demonstrating lack of basic intuition about the world that a three year old would have. Those things perhaps could be caught.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by geysersam on Thursday December 22, 2022 @05:43PM (#34099909)
The data is the bottleneck for the current generation of models. Better models/training strategies could very well change that in the next couple of decades.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by dr_dshiv on Thursday December 22, 2022 @06:10PM (#34100195)
Well, diffusion networks seemed to work because they increased the amount of data by training so many examples of added noise and it’s removal. Some similar approach might be possible with text, too. Or, i guess they will talk to itself, like alpha go played itself.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by rhacker on Thursday December 22, 2022 @06:15PM (#34100257)
The next phase has got to be more and more private data sets that don't exist on the internet: Homes with Alexa, Ok Google, etc.. Beyond that, linkages into a human brain.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by bilsbie on Thursday December 22, 2022 @05:05PM (#34099465)
Yet humans learn on way less language data.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by Enginerrrd on Thursday December 22, 2022 @05:10PM (#34099524)
I disagree.

On pure # of words, sure.

But for humans, language is actually just a compressed version of reality perceived through multiple senses / prediction + observation cycles / model paradigms / scales / contexts/ social cues, etc. and we get full access to the entire thing. So a single sentence is wrapped in orders of magnitude more data.

We also get multiple modes of interconnected feedback. How to describe this? Let me use an analogy. In poker, different properties a player has statistically take different amounts of data to reach convergence: Some become evident in 10s of hands, some take 100s of hands, and some take 1000s, and some even take 10,000s before you get over 90% confidence. ....And yet, if you let a good player see your behavior on just one single hand that goes to showdown, a good human player will be able to estimate your playing style, skill, and where your stats will converge to with remarkable accuracy. They get to see how you acted pre-flop, on the flop, turn, and river, with the rich context of position, pot-size, and what the other players were doing during those times, along with the stakes and location you're playing at, what you're wearing, how you move and handle your chips, etc. etc.

Re:Do Simpler Machine Learning Models Exist and How Can We F... by numpad0 on Thursday December 22, 2022 @05:21PM (#34099657)
We also eat. It feels to me that better food with divergent micronutrients has positive performance implications. Maybe I’m just schizophrenic, but to me it just feels that way.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by fshbbdssbbgdd on Thursday December 22, 2022 @05:39PM (#34099861)
Try training the model with free-range, locally-sourced electrons.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by marcosdumay on Thursday December 22, 2022 @05:30PM (#34099750)
It's less data anyway. There's no way to add up the data a person senses and get into a volume any similar to what's on the internet.

But it may be better data.

Re:Do Simpler Machine Learning Models Exist and How Can We F... by thom on Thursday December 22, 2022 @06:03PM (#34100118)
To quantify this a little, the human sensory system has been estimated to generate on the order of 11m bits per second of data. So 1-2 megabytes a second for most of your life. That’s probably in the region of a day of YouTube. But it’s clear that a lot of human cognition is directed towards novelty, and humans are able to run experiments interactively, not just consume data (see schemas in child development etc).

So, you take your baby AI and instead of training it just on a static corpus, you put it in a simulator. And then when you again hit a wall where you conclude data is the problem, you give them a robot body and plug them into the internet bidirectionally. Some would argue this would be a mistake.

Re:Do Simpler Machine Learning Models Exist and How Can We F... by katvan on Thursday December 22, 2022 @06:28PM (#34100377)
somehow we quickly can apply other models to make predictions on new unrelated datasets. I guess that's where stupidity and creativity come from.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by edgyquant on Thursday December 22, 2022 @07:13PM (#34100760)
Language is a tiny way of encapsulating the vivid imagery humans contain about any given situation. The issue with ML models is they are very specific, a human baby collections 1-2 years worth of visual, auditory, etc data before it begins to use that in a meaningful way. Every thing a baby does is a reinforcement session and every moment it is training its neural networks. This doesn’t even get into sleep: which is where connections are solidified in an abstract way.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by lovecg on Friday December 23, 2022 @02:10AM (#34103080)
Humans benefit from billions of years of pre-trained hardware. It will be interesting to see if we can match that somehow.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by wincy on Thursday December 22, 2022 @05:34PM (#34099806)
Couldn’t we start a website that just has humans tag stuff for machine learning, and make that tagged data set open? Does such a thing exist? I’ve heard the issues with Stable Diffusion and others is that the LAION-5B dataset is kind of terrible quality.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by PoignardAzur on Thursday December 22, 2022 @06:30PM (#34100395)
SurgeAI's business model is to crowdsource these kinds of datasets, some of which it has release in the open.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by nl on Thursday December 22, 2022 @06:57PM (#34100628)
> I’ve heard the issues with Stable Diffusion and others is that the LAION-5B dataset is kind of terrible quality.

This is mostly wrong.

It's possible to get higher quality results in specific domains by fine-tuning on carefully annotated datasets. However these higher quality results wouldn't be possible without the vast pre-training on the huge dataset LAION-5B provides.

> humans tag stuff for machine learning, and make that tagged data set open? Does such a thing exist?

Yes, there are lots. The size of LAION-5B is the innovation here.

Re:Do Simpler Machine Learning Models Exist and How Can We F... by tabtab on Thursday December 22, 2022 @06:24PM (#34100343)
Re: similar to genetic programming

A genetical algorithm was also what I was thinking of. One could devise some kind of symbolic (textual) way to represent a wiring/circuit diagram (graph) and evolve the most efficient "learner" using mutation and cross-breeding (e-sex). The earliest GA I read about used Lisp dicing.

As far as "easiest" AI for humans to work with, "Factor Tables" may be a way:

https://github.com/RowColz/AI

AI tuning then becomes more like accounting instead of a lab with Doc Brown. Factor Tables are much easier to analyze, debug, and modularize than neural nets.

Re:Do Simpler Machine Learning Models Exist and How Can We F... by joaogui1 on Friday December 23, 2022 @04:32AM (#34103894)
There's been work combining GA's and Architecture search for neural networks, the main keyword to search is NeuroEvolution, with NEAT being one of the first "good" algorithms for that (though scaling it up is hard)
Re:Do Simpler Machine Learning Models Exist and How Can We F... by albertzeyer on Thursday December 22, 2022 @06:36PM (#34100448)
> "data, not size, is the currently active constraint on language modeling performance"

This is a bit incomplete. It goes both ways. Right in the abstract, it says:

"the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled"

You are probably referring to the statement:

"We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant."

So GPT3 and co would perform better with more data, but only up to a certain point. At that point, you would also need to scale up the model size again to get better performance.

But also, GPT3 and co would perform better when they would be scaled down a bit, when you keep the same training data. That is actually what the Chinchilla paper does. Their Chinchilla model is smaller than GPT3 and others, trained with same amount of data (as Gopher), and performs better.

I don't really see a problem in scaling up models even more. This paper just says that you should also scale up the training tokens equally.

As far as I remember, this paper does not quite address whether you can use the same training data twice. I think all statements assume that every training token is only used once.

Re:Do Simpler Machine Learning Models Exist and How Can We F... by joaogui1 on Friday December 23, 2022 @04:26AM (#34103865)
Chinchilla is trained with substantially more tokens then Gopher, 1.4T Vs 300B
Re:Do Simpler Machine Learning Models Exist and How Can We F... by jacquesm on Thursday December 22, 2022 @08:43PM (#34101384)
And you can bet that any file timestamped prior to GPT being released will carry extra weight in the future.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by DenisM on Thursday December 22, 2022 @10:19PM (#34102010)
It’s kind of like pre-nuclear steel that way.

https://en.m.wikipedia.org/wiki/Low-background_steel

Re:Do Simpler Machine Learning Models Exist and How Can We F... by jacquesm on Thursday December 22, 2022 @10:25PM (#34102036)
Ah great analogy, yes, that is exactly what it is.
Re:Do Simpler Machine Learning Models Exist and How Can We F... by jacquesm on Thursday December 22, 2022 @10:10PM (#34101953)
I'm reminded of the way computer chess has evolved. It wasn't all that long ago that you needed rooms full of all kinds of special purpose power hungry hardware and then suddenly all that was gone and the strongest programs in the world fit in your pocket on a device using as much power as a small lightbulb. An extra level of understanding of the problem domain achieved through 'slow' methods can be an enormous difference in terms of practical applications.