mysterhawk
Re:Ask HN: Who wants to be hired? (May 2026)
on Wednesday May 06, 2026 @08:00AM (#48035220)
Location: Italy
Remote: Yes
Willing to relocate: No
Technologies: Rust,Python,Java-spring,TypeScript,React,Html,CSS,Nim,C,C++,PHP
Resumé/CV: https://drive.google.com/file/d/1F_Fp3pKusTQtVBgcxYF6h_2leOz... or https://www.linkedin.com/in/antonio-gagliarducci
Email: [email protected]
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge Devices [pdf]
on Thursday October 03, 2024 @12:09PM (#41732016)
With this model:
https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF
Man I need to test the q8 version with llamafiles optimizations, it would be so nice to host it locally with the new ryzens, it could maybe fit my 96GB of ram
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge Devices [pdf]
on Thursday October 03, 2024 @12:06PM (#41731985)
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge Devices [pdf]
on Thursday October 03, 2024 @12:05PM (#41731977)
I could run at 1token/s on qwen 2.5 72b q4_k_m on my i7 8750h + alot of ram XD
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge Devices [pdf]
on Thursday October 03, 2024 @12:04PM (#41731968)
And everything runs way faster on cpu like that
Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge Devices [pdf]
on Thursday October 03, 2024 @12:04PM (#41731967)
You can already do it with llamafile, checkout the project, it lets you convert a .gguf model in a portable executable