Ask HN: Best practice to use Llama 3 8B on production server

Posted by andrew_zhong on Friday April 19, 2024 @02:48PM
from the ask-hn dept.

The new Llama 3 8b in on par with 22b models, better but could be 10x cheaper than GPT3.5

AI builders, if you are using Llama 3 in backend, where do you host it or what API do you use? (For production usecases with good speed and rate limits close to ChatGPT or Claude)

-AWS sagemaker

-Self host on cloud GPUs

-Replicate API (just found them, 0.05/1m token, legit?)

-AWS bedrock (seems pricy)

-Others - pls comment

Any feedback is welcome!

( 1 comment | 3 points | on HN )

Re:Ask HN: Best practice to use Llama 3 8B on production server by whereismyacc on Thursday April 25, 2024 @08:18AM (#40156581)

$0.05 is per million token input, it's $0.25 for output tokens

[ Parent | on HN ]