Re:A single function Jev-like wrapper for LLMs, including vi...
by dist-epoch on Saturday September 26, 2026 @07:17AM
(#49855443)
Jev claims 70-500 ms latency, including for the first request. This requires some clever engineering at least, which will take a little to duplicate.
Maybe first request is unbatched, to have fast prefill, and the subsequent ones are batched.
They also don't restrict your prompt. You can have a dumb one, where you put the variable data at the front, and the details on how to process it at the back, thus you bust the user-part of the KV cache every request.
Maybe first request is unbatched, to have fast prefill, and the subsequent ones are batched.
They also don't restrict your prompt. You can have a dumb one, where you put the variable data at the front, and the details on how to process it at the back, thus you bust the user-part of the KV cache every request.