News for hackers. Stuff that matters.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by griomnib on Saturday December 28, 2024 @02:10PM (#42533584)
Funny thing is you can train a small BERT model to detect queries that are in categories that aren’t ready for AI “answers” with like .00000001% of the energy of an LLM.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hansvm on Saturday December 28, 2024 @03:49PM (#42534590)
That's (obviously) a bit of an exaggeration. BERT is just another transformer architecture. Cut down from ~100 layers to 1, ~1k dimensions to ~10, and ~10k tokens to 100, and you're only 1e6 faster / more efficient, still a factor of 10k greater than your estimate and also too small to handle the detection you're describing with any reasonable degree of accuracy.
Re:Google's Results Are Infested, Open AI Is Using Their Pla... by griomnib on Saturday December 28, 2024 @05:17PM (#42535292)
I literally have DistilBERT models that can do this exact task in ~14ms on an NVIDIA A6000. I don’t know the precise performance per watt, but it’s really fucking low.

I use LLM to help with training data as they are great at zero shot, but after the training corpora is built a small, well trained, model will smoke an LLM in classification accuracy and are way faster - which means you can get scale and low carbon cost.

In my personal opinion there is a moral imperative to use the most efficient models possible at every step in a system design. LLM are one type of architecture and while they do a lot well, you can use a variety of energy efficient techniques to do discrete tasks much better.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by hansvm on Saturday December 28, 2024 @05:53PM (#42535552)
Thanks for providing a concrete model to work with. Compared to GPT3.5, the number you're looking for is ~0.04%. I pointed out the napkin math because 0.00000001% was so obviously wrong even at a glance that it was hurting your claim.

And, yes, purpose-built models definitely have their place even with the advent of LLMs. I'm happy to see more people working on that sort of thing.

Re:Google's Results Are Infested, Open AI Is Using Their Pla... by griomnib on Saturday December 28, 2024 @06:53PM (#42536010)
I applaud you doing the math! Proves you aren’t an LLM :-D