Re:Mistral 7B, the most powerful language model for its size...
by brucethemoose2 on Wednesday September 27, 2023 @12:52PM
(#37677509)
Try MLC-LLM. Its not as bad as you'd think.
In the future(?), they will probably use the AI blocks instead of the GPU, which are very low power.
Even running a quantized and optimized LLM on a smartphone would kill battery life at minimum.