Forget ChatGPT: why researchers now run small AIs on their laptops

Re:Forget ChatGPT: why researchers now run small AIs on thei... by jonnycomputer on Monday September 23, 2024 @11:05AM (#41626949)

RAM and what GPU you have are big determinants of how fast it will run, and how smart a model you can run. A large amount of RAM and GPU memory is required for larger models without significant slowdown because its much faster if it can keep the entire model in memory. Small models range from 3-8 gigabytes, but a 70B parameter model will be 30-50 gigabytes.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by neop1x on Monday September 23, 2024 @06:41PM (#41631356)

I am running 70B models on M2 Max with 96 GB of RAM and it works very well. As HW evolves, it will become a standard

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by creata on Tuesday September 24, 2024 @03:21AM (#41633964)

Out of curiosity, what degree of quantization are you applying to these 70B models?

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by neop1x on Tuesday September 24, 2024 @05:35AM (#41634778)

Q4_K_S. While not as good as top commercial models like chatgpt, they are still quite capable and I like that there are also uncensored/abliterated models like Dolphin.

[ Parent | on HN ]