Re:What's new in Llama 2 and how to run it locally
by simonw on Monday August 07, 2023 @02:12AM
(#37030782)
If you want to try Llama 2 on a Mac and have Homebrew (or Python/pip) you may find my LLM CLI tool interesting: https://simonwillison.net/2023/Aug/1/llama-2-mac/
[ Parent
| on HN ]
Re:What's new in Llama 2 and how to run it locally
by petulla on Monday August 07, 2023 @02:20AM
(#37030858)
What's the inference time without gpu?
Re:What's new in Llama 2 and how to run it locally
by lm2s on Monday August 07, 2023 @02:36AM
(#37030975)
It might the time mentioned at the bottom of the page since the author isn't sure that the GPU is being used:
>How to speed this up—right now my Llama prompts often take 20+ seconds to complete.
Re:What's new in Llama 2 and how to run it locally
by bravura on Monday August 07, 2023 @07:20AM
(#37032971)
Does it support Metal / MPS acceleration?
Re:What's new in Llama 2 and how to run it locally
by SOLAR_FIELDS on Monday August 07, 2023 @08:51AM
(#37033716)
I’ve gotten mine running with FastChat - they have a Metal/MPS option.
Sadly 7b is not very good for SQL tasks. I think even with RAG it would struggle.