News for hackers. Stuff that matters.
Re:Mistral 7B, the most powerful language model for its size... by minimaxir on Wednesday September 27, 2023 @12:24PM (#37677065)
"usable" is not the same as practical.

Even running a quantized and optimized LLM on a smartphone would kill battery life at minimum.

Re:Mistral 7B, the most powerful language model for its size... by brucethemoose2 on Wednesday September 27, 2023 @12:52PM (#37677509)
Try MLC-LLM. Its not as bad as you'd think.

In the future(?), they will probably use the AI blocks instead of the GPU, which are very low power.