Re:What happens if we remove 50 percent of Llama?
by mewpmewp2 on Monday December 02, 2024 @09:40PM
(#42302620)
Don't we still have a lot of video, and other non text real world data to go with? Feels like a possible potential break from there.
Re:What happens if we remove 50 percent of Llama?
by visarga on Tuesday December 03, 2024 @09:40AM
(#42306543)
Generally speaking text only models manage to learn a huge amount about the visual world. So when you put the model train on video it might have less to learn. Video is also less abstract than text, generally. But I am sure we can still extract useful learning from videos, it's probably expensive, but we'll have to do that at some point.
Re:What happens if we remove 50 percent of Llama?
by griomnib on Tuesday December 03, 2024 @11:29AM
(#42307981)
Given how much of the web is ai generated slop now, I think going forward itβs even worse than you suggest.
I have a copy of refined web locally so I have a billion pre-chatgpt documents for my long term use.
Going forward we will accumulate truly useful data at a linear growing rate. This fundamentally breaks the scaling game. If your model and compute expand exponentially but your training data only linearly, the efficiency won't be the same.
Synthetic data might help us pad up the training sets, but the most promising avenue I think is to use user-LLM chat logs. Those logs contain real world grounding and human in the loop. Millions of humans doing novel tasks. But that only scales linearly with time, as well.
No way around it - we only once had the whole internet for the first time in the training set. After that it's linear time.