Re:SSE sucks for transporting LLM tokens
by ivan_gammel on Saturday December 13, 2025 @01:54PM
(#46256929)
I don’t get it. Client generates UUID for prompt, PUTs the prompt with this UUID on server. Server caches the generated output for reasonable time, so that subsequent PUTs get 200 instead of 201. Transport protocol failures then do not matter. If response isn’t 4x, just retry.
[ Parent
| on HN ]
Re:SSE sucks for transporting LLM tokens
by riskable on Saturday December 13, 2025 @02:03PM
(#46256986)
The way the current architecture works—as far as I know—is your assumed "server caches the generated output" step doesn't exist. What you get in your output is streamed directly from the LLM to your client. Which is, in theory, the most efficient way to do it.
That's why LLM outputs that get cut off mid-stream require the end user click the "retry" button and not the, "re-send me that last output" button (which doesn't exist).
I would imagine that a simpler approach would be to simply make the last prompt idempotent... Which would require caching on their servers; something that supposedly isn't happening right now. That way, if the user re-sends the last prompt the server just responds with the same exact output it just generated. Except LLMs often make mistakes and hallucinate things... So re-sending the last prompt and hoping for a better output isn't an uncommon thing.
Soooo... Back to my suggested workaround in my other comment: Pub/sub over WebSockets :D
The user's last prompt can be sent with an idempotency key that changes each time the user initiates a new request. If that's the same, use the cache. If it's new, hit the LLM again.
Re:SSE sucks for transporting LLM tokens
by ivan_gammel on Saturday December 13, 2025 @02:26PM
(#46257158)
The only reason LLM server responds with partial results instead of waiting and returning all at once is UX. It’s just too slow. But the problem of slow bulk responses isn’t unique for LLM and can be solved within HTTP 1.1 well enough. Doesn’t have to be the same server, can be a caching proxy in front of it. Any privacy concerns can be addressed by giving the user opportunity to tell server to cache/not to cache (can be as easy as submitting with PUT vs POST requests)
Re:SSE sucks for transporting LLM tokens
by debazel on Saturday December 13, 2025 @02:11PM
(#46257043)
But adding caching to SSE is trivial compared to completely changing your transfer protocol, so why wouldn't you just do that instead?
Re:SSE sucks for transporting LLM tokens
by verdverm on Saturday December 13, 2025 @03:19PM
(#46257601)
It's likely better to just store them in the database, it could be hours or days later that I want to look at the session again. If you're going to want a database for conversation anyway, then it doesn't make sense to cache and query individual messages
Re:SSE sucks for transporting LLM tokens
by ivan_gammel on Saturday December 13, 2025 @03:57PM
(#46257976)
A database can be the implementation of choice for the cache. But not all use cases do require long-term storage like that.
Re:SSE sucks for transporting LLM tokens
by verdverm on Saturday December 13, 2025 @05:28PM
(#46258804)
yup, the ADK framework I use handles all that bookkeeping for me, has a few options built in, and pretty easy to add new implementations. I'm extending it to attach filesys+exe changes during a session, persisted as container layers via Dagger
Re:SSE sucks for transporting LLM tokens
by imsh4yy on Saturday December 13, 2025 @02:03PM
(#46256992)
Yep, came here expecting to read an interesting take on why SSE sucks or a better alternative, but this just reads like "skill issue." A term I very much dislike but seems appropriate here.
Re:SSE sucks for transporting LLM tokens
by ivan_gammel on Saturday December 13, 2025 @02:35PM
(#46257218)
Significant part of relatively new technology stacks and tech slang is “skill issue”. A lot of problems were already solved or at least analyzed 40-20 years ago and hardly need to be re-invented, maybe just modernized.