Re:Performance of llama.cpp on Apple Silicon A-series
by discobot on Tuesday December 19, 2023 @07:06PM
(#38703762)
regarding the prompt processing and token generation you are correct
it makes sense to benchmark them infependently since prompt processing is done in parralel for each token and is compute bound and token generation is sequential and bound by memory banwidth
it makes sense to benchmark them infependently since prompt processing is done in parralel for each token and is compute bound and token generation is sequential and bound by memory banwidth