Re:Making AMD GPUs competitive for LLM inference
by paulmd on Thursday August 10, 2023 @12:00PM
(#37077876)
is 70B not commercially useful because of the model storage requirements, or total inference performance, or additional memory per session that's inferencing, or what?
is the output better such that it's desirable, or is this just a case of "too much performance hit for a marginal gain"?
is the output better such that it's desirable, or is this just a case of "too much performance hit for a marginal gain"?