A single function Jev-like wrapper for LLMs, including vision models

Re:A single function Jev-like wrapper for LLMs, including vi... by imtringued on Saturday September 26, 2026 @06:23AM (#49855111)

They can't overturn the economics of attention by restricting themselves to a single token output.

Sure they are no longer memory bandwidth bound thanks to that but someone could add a similar projector to a conventional model, train with a Jev style dataset and call it a day.

Whatever they are doing on inputs must either mean they intentionally chose a Mamba successor or they suffer from the same compute costs as everyone else.

[ Parent | on HN ]

Re:A single function Jev-like wrapper for LLMs, including vi... by dist-epoch on Saturday September 26, 2026 @07:17AM (#49855443)

Jev claims 70-500 ms latency, including for the first request. This requires some clever engineering at least, which will take a little to duplicate.

Maybe first request is unbatched, to have fast prefill, and the subsequent ones are batched.

They also don't restrict your prompt. You can have a dumb one, where you put the variable data at the front, and the details on how to process it at the back, thus you bust the user-part of the KV cache every request.

[ Parent | on HN ]