News for hackers. Stuff that matters.

A comment on Show HN: Khoj – Chat Offline with Your Second Brain Using Llama 2.

Re:Show HN: Khoj – Chat Offline with Your Second Brain Using... by mavsman on Sunday July 30, 2023 @08:37PM (#36937623)
Really cool to see this! Local is the real future of AI.

I got really excited about this and fired it up on my petite little M2 Macbook Air only for it to grind it to a halt. Think the old days when you had a virus on your PC and you'd move the mouse then wait 45 seconds to see the cursor move. It honestly made me feel nostalgic. I guess I have to taper performance expectations with this Air, though this is the first time it's happened.

[ Parent | on HN ]
Re:Show HN: Khoj – Chat Offline with Your Second Brain Using... by jmorgan on Sunday July 30, 2023 @11:48PM (#36938617)
How much memory do you have in your Macbook? The 7B models seem to work well with at least 16GB of unified memory, but I’ve seen Macs with 8GB really struggle.
[ Parent | on HN ]
Re:Show HN: Khoj – Chat Offline with Your Second Brain Using... by mavsman on Monday July 31, 2023 @09:43AM (#36942331)
Indeed it's just a poor little 8GB of RAM.
[ Parent | on HN ]
Re:Show HN: Khoj – Chat Offline with Your Second Brain Using... by quickthrower2 on Monday July 31, 2023 @12:52AM (#36938934)
Just wait 10 years when computers have a dedicated AI-PU and you don't have to worry about freezing anything up to talk to your bot.
[ Parent | on HN ]
Re:Show HN: Khoj – Chat Offline with Your Second Brain Using... by kridsdale1 on Monday July 31, 2023 @01:02PM (#36945593)
The M1 line of Macs does have that unit.
[ Parent | on HN ]
Re:Show HN: Khoj – Chat Offline with Your Second Brain Using... by joaogui1 on Monday July 31, 2023 @02:25PM (#36946770)
I wonder if edge tpus (sold here https://coral.ai/products/) could help, though I guess the community would have to optimize for them instead of for standard hardware
[ Parent | on HN ]