Performance of llama.cpp on Apple Silicon A-series

Re:Performance of llama.cpp on Apple Silicon A-series by cgearhart on Tuesday December 19, 2023 @06:49PM (#38703611)

What’s the definition of “prompt processing” vs “token generation”?

Is that separately comparing the time it takes to preprocess the input prompt (prompt_length / pp_token_rate = time_to_first_token) and then the token generation rate is the time for each successive token?

I also see something about bs batch size. Is batching relevant for a locally run model? (Usually you only have one prompt at a time, right?)

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by andy99 on Tuesday December 19, 2023 @07:10PM (#38703792)

It says at the top that prompts are processed with a batch size of 512, while new tokens can only be generated one at a time with batch size of 1.

Prompt processing doesn't actually need to compute logits for each new token, just cache the KV values and so is much faster than actual inference.

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by idonotknowwhy on Tuesday December 19, 2023 @07:10PM (#38703786)

Yeah prompt processing is tokenizing the input, which includes the system prompt, chat history and your chat text.

Generation is the response it sends back

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by discobot on Tuesday December 19, 2023 @07:06PM (#38703762)

regarding the prompt processing and token generation you are correct

it makes sense to benchmark them infependently since prompt processing is done in parralel for each token and is compute bound and token generation is sequential and bound by memory banwidth

[ Parent | on HN ]