Life of an inference request (vLLM V1): How LLMs are served efficiently at scale

Re:Life of an inference request (vLLM V1): How LLMs are serv... by animan on Saturday June 28, 2025 @08:10PM (#44409243)

Prefill is part of Inference. It's the first major step where you calculate all the keys and values for the input tokens.

Decode is the next major step where you start generating output tokens one at a time.

Both run on GPUs but have slightly different workloads

1. Prefill has very little I/o from VRAM to HBM and more compute 2. Decode is light on compute but have to I/o the keys and values computed in the prefill stage for every output token

[ Parent | on HN ]

Re:Life of an inference request (vLLM V1): How LLMs are serv... by dist-epoch on Sunday June 29, 2025 @02:27AM (#44410753)

Doesn't decode also need to stream in the whole of the model weights, thus very I/O heavy?

[ Parent | on HN ]

Re:Life of an inference request (vLLM V1): How LLMs are serv... by 0xjunhao on Monday June 30, 2025 @09:26AM (#44423026)

Yes, decoding is very I/O heavy. It has to stream in the whole of the model weights from HBM for every token decoded. However, that cost can be shared between the requests in the same batch. So if the system has more GPU RAM to hold larger batches, the I/O cost per request can be lowered.

[ Parent | on HN ]