how do you do that exactly?
Not the original commenter but I’ve done this too with Pydantic AI (actually the library does it for you). See “Streaming Structured Output” here https://ai.pydantic.dev/output/#streaming-structured-output
Thanks yes! Im aware of structured outputs, llama.cpp has also great support with GBNF and several languages beyond json.
I've been trying to create go/rust ones but its way harder than just json due to all the context/state they carry over
One way is to eagerly call JSON.parse as fragments are coming in. If you also split on json semantic boundaries like quotes/closing braces/closing brackets, you can detect valid objects and start processing them while the stream continues.
Interesting approach! thanks for sharing
Can you offer some detail into why you find this approach useful?
From an outsider's perspective, if you're sending around JSON documents so big that it takes so long to parse them to the point reordering the content has any measurable impact on performance, this sounds an awful lot like you are batching too much data when you should be progressively fetching child resources in separate requests, or even implementing some sort of pagination.
It’s become a thing, even beyond RSCs, and has many practical uses if you stare at the client and server long enough.