News for hackers. Stuff that matters.

A comment on Serving 70B-Scale LLMs Efficiently on Low-Resource Edge Devices [pdf]. Parent comment

Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by mysterhawk on Thursday October 03, 2024 @12:04PM (#41731967)
You can already do it with llamafile, checkout the project, it lets you convert a .gguf model in a portable executable
[ Parent | on HN ]
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by mysterhawk on Thursday October 03, 2024 @12:06PM (#41731985)
https://justine.lol/matmul/

https://www.youtube.com/watch?v=-mRi-B3t6fA

Checkout these articles

[ Parent | on HN ]
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by mysterhawk on Thursday October 03, 2024 @12:04PM (#41731968)
And everything runs way faster on cpu like that
[ Parent | on HN ]
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by mysterhawk on Thursday October 03, 2024 @12:05PM (#41731977)
I could run at 1token/s on qwen 2.5 72b q4_k_m on my i7 8750h + alot of ram XD
[ Parent | on HN ]
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by mysterhawk on Thursday October 03, 2024 @12:09PM (#41732016)
With this model:

https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF

Man I need to test the q8 version with llamafiles optimizations, it would be so nice to host it locally with the new ryzens, it could maybe fit my 96GB of ram

[ Parent | on HN ]