News for hackers. Stuff that matters.
Re:What's new in Llama 2 and how to run it locally by simonw on Monday August 07, 2023 @02:12AM (#37030782)
If you want to try Llama 2 on a Mac and have Homebrew (or Python/pip) you may find my LLM CLI tool interesting: https://simonwillison.net/2023/Aug/1/llama-2-mac/
Re:What's new in Llama 2 and how to run it locally by petulla on Monday August 07, 2023 @02:20AM (#37030858)
What's the inference time without gpu?
Re:What's new in Llama 2 and how to run it locally by lm2s on Monday August 07, 2023 @02:36AM (#37030975)
It might the time mentioned at the bottom of the page since the author isn't sure that the GPU is being used:

>How to speed this up—right now my Llama prompts often take 20+ seconds to complete.

Re:What's new in Llama 2 and how to run it locally by bravura on Monday August 07, 2023 @07:20AM (#37032971)
Does it support Metal / MPS acceleration?
Re:What's new in Llama 2 and how to run it locally by SOLAR_FIELDS on Monday August 07, 2023 @08:51AM (#37033716)
I’ve gotten mine running with FastChat - they have a Metal/MPS option.

Sadly 7b is not very good for SQL tasks. I think even with RAG it would struggle.