Forget ChatGPT: why researchers now run small AIs on their laptops

Re:Forget ChatGPT: why researchers now run small AIs on thei... by albertgoeswoof on Saturday September 21, 2024 @11:16AM (#41610523)

I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying.

If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question.

However I cannot use the online providers for anything remotely sensitive, which is more often than you might think.

Local LLMs are the future, it’s like having your own private Google running locally.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by fsmv on Saturday September 21, 2024 @12:02PM (#41610786)

A small model necessarily is missing many facts. The large model is the one that has memorized the whole internet, the small one is just trained to mimic the big one.

You simply cannot compress the whole internet under 10gb without throwing out a lot of information.

Please be careful about what you take as fact coming from the local model output. Small models are better suited to summarization.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by albertgoeswoof on Saturday September 21, 2024 @02:44PM (#41611827)

I don’t trust anything as fact coming out of these models. I ask it for how to structure solutions, with examples. Then I read the output and research the specifics before using anything further.

I wouldn’t copy and paste from even the smartest minds, nevermind a model output.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by beernet on Sunday September 22, 2024 @08:35AM (#41616602)

> The large model is the one that has memorized the whole internet

This is totally wrong and a potentially dangerous way to think about LLMs. They have no clue about what's factual knowledge and what is not, per design.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by staticman2 on Saturday September 21, 2024 @12:04PM (#41610793)

I'm really curious what you are doing with an LLM that can be solved 80% of the time with a 8b model.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by albertgoeswoof on Saturday September 21, 2024 @02:46PM (#41611831)

It’s mostly how would you solve this programming problem, or reminders on syntax, scaffolding a configuration file etc.

Often it’s a form of rubber duck programming, with a smarter rubber duck.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by skydhash on Saturday September 21, 2024 @05:59PM (#41612946)

All of this can be solved with a 3-20MB PDF file, a 10kb snippet/template file, and a whiteboard.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by rNULLED on Saturday September 21, 2024 @06:46PM (#41613222)

Don’t forget the duct tape

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by remram on Sunday September 22, 2024 @01:34PM (#41618545)

"Dropbox will never work, you can already build such a system yourself quite trivially by getting an FTP account and mounting it locally with curlftpfs"

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by meiraleal on Saturday September 21, 2024 @11:24AM (#41610582)

We need browser and OS level API (mobile) integration to the local LLM.

[ Parent | on HN ]

Re:Forget ChatGPT: why researchers now run small AIs on thei... by evilduck on Saturday September 21, 2024 @02:59PM (#41611912)

Both are in their early stages:

https://developer.chrome.com/docs/ai

https://developer.apple.com/documentation/AppIntents/Integra...

[ Parent | on HN ]