Forget ChatGPT: why researchers now run small AIs on their laptops

Re:Forget ChatGPT: why researchers now run small AIs on thei... by threecheese on Saturday September 21, 2024 @02:28PM (#41611735)

Have you tried a Llamafile? Not sure what platform you are using. From their readme:

  > … by combining llama.cpp with Cosmopolitan Libc into one framework that collapses all the complexity of LLMs down to a single-file executable (called a "llamafile") that runs locally on most computers, with no installation.

Low cost to experiment IMO. I am personally using MacOS with an M1 chip and 64gb memory and it works perfectly, but the idea behind this project is to democratize access to generative AI and so it is at least possible that you will be able to use it.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by narrator on Saturday September 21, 2024 @02:49PM (#41611853)

With 64GB can you run the 70B size llama models well?

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by credit_guy on Saturday September 21, 2024 @03:37PM (#41612161)

No, you can't. I have 128 GB and a 70B llamafile is unusable.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by threecheese on Saturday September 21, 2024 @04:37PM (#41612524)

I should have qualified the meaning of “works perfectly” :) No 70b for me, but I am able to experiment with many quantized models (and I am using a Llama successfully, latency isn’t terrible)

[ Parent | on HN ]