News for hackers. Stuff that matters.
Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by junrushao1994 on Sunday April 30, 2023 @12:23PM (#35764060)
Thanks for sharing! It's definitely a bit painstaking to get a real-world LLM running at all on an iPhone due to memory constraint. It's also quite compute-intense as it has 7B parameters, but we are glad that it's generating texts at reasonable speed!

The model we are using is a quantized Vicuna-7b, which I believe is one of the best open-sourced models. Hallucination is a problem to all LLMs, but I believe research on model side would gradually alleviate this problem :-)

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by Fr0styMatt88 on Sunday April 30, 2023 @05:01PM (#35766594)
WizardLM-7b would be a fantastic model to try out as well. Though that might be out of date tomorrow (or already!) given how many models are being released at the moment :)