Deploying Llama3 70B on AWS – GPU Requirement, Cost and Step-by-Step Guide

Posted by JJneid on Thursday June 13, 2024 @06:31PM
from the slashml.com dept.

https://www.slashml.com/blog/deploying-llama-3-70b-on-aws

( 5 comments | 2 points | on HN )

Re:Deploying Llama3 70B on AWS – GPU Requirement, Cost and S... by rini17 on Thursday June 13, 2024 @07:25PM (#40675987)

Note that quantized versions of llama3 70B can be ran on CPU on much cheaper server. I am personally using it via llama.cpp on bare metal 6-core Xeon CPU with 128M RAM for ~50 euro monthly.

[ Parent | on HN ]

Re:Deploying Llama3 70B on AWS – GPU Requirement, Cost and S... by JJneid on Thursday June 13, 2024 @09:02PM (#40676549)

Is inference speed an issue for you?

[ Parent | on HN ]

Re:Deploying Llama3 70B on AWS – GPU Requirement, Cost and S... by rini17 on Friday June 14, 2024 @04:05AM (#40678689)

Sufficient for fluent conversation.

[ Parent | on HN ]

Re:Deploying Llama3 70B on AWS – GPU Requirement, Cost and S... by JJneid on Thursday June 13, 2024 @09:05PM (#40676565)

usually performance takes a hit with quantization. are you getting quality responses?

[ Parent | on HN ]

Re:Deploying Llama3 70B on AWS – GPU Requirement, Cost and S... by rini17 on Friday June 14, 2024 @04:04AM (#40678682)

Since llama3, yes, quite satisfying.

[ Parent | on HN ]