News for hackers. Stuff that matters.

A comment on (Databricks cofounder) How Is ChatGPT’s Behavior Changing over Time?. Parent comment

Re:(Databricks cofounder) How Is ChatGPT’s Behavior Changing... by Sosh101 on Wednesday July 19, 2023 @12:16AM (#36782058)
> and $10,000+ of compute hardware per inference session.

What hardware would you need to run it at home?

> Full power Llama 2 still requires multiple kilowatts of electricity

Assume you mean kilowatt-hours? (sorry to be pedantic)

[ Parent | on HN ]
Re:(Databricks cofounder) How Is ChatGPT’s Behavior Changing... by ramesh31 on Wednesday July 19, 2023 @04:48AM (#36783611)
>What hardware would you need to run it at home?

Step 1: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma...

Step 2: https://github.com/ggerganov/llama.cpp

Step 3: you're welcome

[ Parent | on HN ]
Re:(Databricks cofounder) How Is ChatGPT’s Behavior Changing... by Sosh101 on Wednesday July 19, 2023 @09:09AM (#36785675)
That's very helpful, thank you.
[ Parent | on HN ]
Re:(Databricks cofounder) How Is ChatGPT’s Behavior Changing... by avion23 on Wednesday July 19, 2023 @10:01AM (#36786489)
> and $10,000+ of compute hardware per inference session.

That is not true. A common macbook with lots of RAM (>32GB) is enough. Or any x86 computer with lots of RAM. llama.cpp is CPU only and quite fast

[ Parent | on HN ]