News for hackers. Stuff that matters.

A comment on The Era of 1-bit LLMs: ternary parameters for cost-effective computing. Parent comment

Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by pennomi on Wednesday February 28, 2024 @11:35AM (#39540297)
500 t/s is uncomfortably fast to me. Generating high quality answers at speeds faster than I can read is the point at which I feel like LLMs are magic.

I’m glad people are doing it though, and I’ll happily adapt to accessing inference at that speed.

[ Parent | on HN ]
Re:The Era of 1-bit LLMs: ternary parameters for cost-effect... by azinman2 on Wednesday February 28, 2024 @01:43PM (#39542030)
That's important for new applications to emerge where this happens on lots of data. You can't run LLMs at scale on tasks like Google might (every webpage) when the cost of each document is so high to process. Interactive chatbots are just the tip.
[ Parent | on HN ]