News for hackers. Stuff that matters.

A comment on Ask HN: Who is using small OS LLMs in production?. Parent comment

Re:Ask HN: Who is using small OS LLMs in production? by boredumb on Wednesday August 02, 2023 @09:17AM (#36970465)
OpenAI costs money and llama 2 i'm able to run on my GPU so at least for development purposes at the moment it's ""free"" for me to experiment with - mind you this is a side project with zero funding outside of myself, ymmv if you have access to funding to a point of making OpenAI tokens totally disposable.
[ Parent | on HN ]
Re:Ask HN: Who is using small OS LLMs in production? by LewisVerstappen on Wednesday August 02, 2023 @09:21AM (#36970525)
Ah okay, yeah I was curious how much you'd pay running it on a cloud server in a production-type scenario. Thanks
[ Parent | on HN ]
Re:Ask HN: Who is using small OS LLMs in production? by boredumb on Wednesday August 02, 2023 @09:53AM (#36970927)
I have a vultr account I just logged into and checked, I could rent enough GPU memory to run llama 2 for ~180$ a month from my very quick look - so if I make more than 27000 requests with a 4k token payload I will break even - otherwise I could be using openAIs API and be making out better.

EDIT** im sorry that was gpt3.5 - with gpt4 it would be 1000 requests before I broke even at 180$

[ Parent | on HN ]