News for hackers. Stuff that matters.

A comment on Forget ChatGPT: why researchers now run small AIs on their laptops. Parent comment

Re:Forget ChatGPT: why researchers now run small AIs on thei... by neop1x on Monday September 23, 2024 @06:41PM (#41631356)
I am running 70B models on M2 Max with 96 GB of RAM and it works very well. As HW evolves, it will become a standard
[ Parent | on HN ]
Re:Forget ChatGPT: why researchers now run small AIs on thei... by creata on Tuesday September 24, 2024 @03:21AM (#41633964)
Out of curiosity, what degree of quantization are you applying to these 70B models?
[ Parent | on HN ]
Re:Forget ChatGPT: why researchers now run small AIs on thei... by neop1x on Tuesday September 24, 2024 @05:35AM (#41634778)
Q4_K_S. While not as good as top commercial models like chatgpt, they are still quite capable and I like that there are also uncensored/abliterated models like Dolphin.
[ Parent | on HN ]