Ask HN: Who is using small OS LLMs in production?

Re:Ask HN: Who is using small OS LLMs in production? by ramesh31 on Wednesday August 02, 2023 @09:40AM (#36970746)

Running llama-2-7b-chat at 8 bit quantization, and completions are essentially at GPT-3.5 levels on a single 4090 using 15gb VRAM. I don't think most people realize just how small and efficient these models are going to become.

[ Parent | on HN ]

Re:Ask HN: Who is using small OS LLMs in production? by jeanloolz on Wednesday August 02, 2023 @09:49AM (#36970883)

7B or 70B?

[ Parent | on HN ]

Re:Ask HN: Who is using small OS LLMs in production? by ramesh31 on Wednesday August 02, 2023 @10:01AM (#36971027)

>7B or 70B?

7B 8bit GGML running on a single 4090 with llama.cpp. It's hard to overstate the massive jump in capability between llama 1 and 2.

[ Parent | on HN ]

Re:Ask HN: Who is using small OS LLMs in production? by rimeice on Wednesday August 02, 2023 @10:09AM (#36971125)

Are you hosting that somewhere? If so, how much does that cost and do you have concurrent users?

[ Parent | on HN ]

Re:Ask HN: Who is using small OS LLMs in production? by ramesh31 on Wednesday August 02, 2023 @10:19AM (#36971259)

>Are you hosting that somewhere?

Tensordock. RTX4090 instances are ~$0.50/hr and can handle 3/4 concurrent users each.

[ Parent | on HN ]