News for hackers. Stuff that matters.
Re:Google claims new AI training tech is 13x faster, 10x mor... by seeknotfind on Monday July 08, 2024 @08:09AM (#40904652)
Means 13x bigger models? I hear the data is the limit, but I don't believe it.
Re:Google claims new AI training tech is 13x faster, 10x mor... by raxxorraxor on Monday July 08, 2024 @08:40AM (#40904893)
No, it would mean 13x times smaller models where training data is better curated.

Ironically I think this is a step towards Clip again, which Google already improved with SigLIP but now returns to introduce a global or "batch" perspective. At least that is how I read it with my very amateur understanding.

Re:Google claims new AI training tech is 13x faster, 10x mor... by seeknotfind on Monday July 08, 2024 @02:11PM (#40907759)
Holding the model needs constant, but the real limit is $$$, so it'll mean more model for the same money.
Re:Google claims new AI training tech is 13x faster, 10x mor... by 343242dfsdf on Monday July 08, 2024 @09:57AM (#40905488)
>the data is the limit

Most people don't know exactly how the dataset is "fed" to the training pipeline, but with the current state-of-the-art you can say the feeding is like when a human reads aloud a text, not re-reading ever, not a single word.

And then you're asked about what words you've read most often, their order and how many times they appeared in the text. Then, with those numbers you just gave, some probabilities are calculated and anotated, and there you get a "token".

There are obvious improvements plausible to be applied to that basic processing, and most are being applied already, but there's plenty of room for evern further improvement apparently.

Claude says that the previous text could be described like "a simplified metaphor of model training", so you're warned about simplicity.

Re:Google claims new AI training tech is 13x faster, 10x mor... by tetris11 on Monday July 08, 2024 @08:17AM (#40904731)
But more power still, yes?

10x more efficient power usage than what it would have been for a 13x faster/bigger model, but still more power usage for the new training tech than in the older revision.

Re:Google claims new AI training tech is 13x faster, 10x mor... by MattPalmer1086 on Monday July 08, 2024 @12:56PM (#40907081)
No, it says it is 13 times faster, and 10x more power efficient.

So you could train a model 10 times bigger for the same power usage (assuming it scales linearly), and it would be a bit faster to do that as well.

Or you could train a model 13 times bigger for the same speed, and use 10x less power than if you hadn't.

Re:Google claims new AI training tech is 13x faster, 10x mor... by sva_ on Monday July 08, 2024 @08:50AM (#40904959)
Re:Google claims new AI training tech is 13x faster, 10x mor... by xnx on Monday July 08, 2024 @09:25AM (#40905206)