News for hackers. Stuff that matters.

A comment on Deploying Llama3 70B on AWS – GPU Requirement, Cost and Step-by-Step Guide.

Re:Deploying Llama3 70B on AWS – GPU Requirement, Cost and S... by rini17 on Thursday June 13, 2024 @07:25PM (#40675987)
Note that quantized versions of llama3 70B can be ran on CPU on much cheaper server. I am personally using it via llama.cpp on bare metal 6-core Xeon CPU with 128M RAM for ~50 euro monthly.
[ Parent | on HN ]
Re:Deploying Llama3 70B on AWS – GPU Requirement, Cost and S... by JJneid on Thursday June 13, 2024 @09:02PM (#40676549)
Is inference speed an issue for you?
[ Parent | on HN ]
Re:Deploying Llama3 70B on AWS – GPU Requirement, Cost and S... by rini17 on Friday June 14, 2024 @04:05AM (#40678689)
Sufficient for fluent conversation.
[ Parent | on HN ]
Re:Deploying Llama3 70B on AWS – GPU Requirement, Cost and S... by JJneid on Thursday June 13, 2024 @09:05PM (#40676565)
usually performance takes a hit with quantization. are you getting quality responses?
[ Parent | on HN ]
Re:Deploying Llama3 70B on AWS – GPU Requirement, Cost and S... by rini17 on Friday June 14, 2024 @04:04AM (#40678682)
Since llama3, yes, quite satisfying.
[ Parent | on HN ]