News for hackers. Stuff that matters.

A comment on Forget ChatGPT: why researchers now run small AIs on their laptops. Parent comment

Re:Forget ChatGPT: why researchers now run small AIs on thei... by narrator on Saturday September 21, 2024 @02:49PM (#41611853)
With 64GB can you run the 70B size llama models well?
[ Parent | on HN ]
Re:Forget ChatGPT: why researchers now run small AIs on thei... by credit_guy on Saturday September 21, 2024 @03:37PM (#41612161)
No, you can't. I have 128 GB and a 70B llamafile is unusable.
[ Parent | on HN ]
Re:Forget ChatGPT: why researchers now run small AIs on thei... by threecheese on Saturday September 21, 2024 @04:37PM (#41612524)
I should have qualified the meaning of “works perfectly” :) No 70b for me, but I am able to experiment with many quantized models (and I am using a Llama successfully, latency isn’t terrible)
[ Parent | on HN ]