Re:Progressive JSON
by andrewingram on Sunday June 01, 2025 @07:53AM
(#44150218)
For the use case of streaming data for UI, I don’t think empty arrays and nulls are sufficient information. At any moment during the stream, you need the ability to tell what data is pending.
If pending arrays are just returned as empty arrays, how do I know if it’s empty because it’s actually empty, or empty because it’s pending?
GraphQL’s streaming payloads try to get the best of both worlds, at any point in time you have a valid payload according the GraphQL schema - so it’s possible to render some valid UI, but it also communicates what paths contain pending data, and then subsequent payloads act as patches (though not as sophisticated as Mendoza’s).
Re:Progressive JSON
by atombender on Sunday June 01, 2025 @08:01AM
(#44150264)
As I commented in https://news.ycombinator.com/item?id=44150238, all you need is a way to express what is pending, which can be done using JSON key paths.
Of course, you could do it in-band, too:
{"comments": {"state": "pending", "values": []}}
Re:Progressive JSON
by andrewingram on Sunday June 01, 2025 @08:03AM
(#44150278)
Yes, hence the last paragraph in my comment :)
They want to know where the holes are so that they can show a loading state.
Re:Progressive JSON
by atombender on Sunday June 01, 2025 @07:58AM
(#44150238)
You don't need templating ($1 etc.) for that as long as you can describe the holes somehow, which can be done out-of-band.
If we imagine a streaming protocol of key/value pairs that are either snapshots or deltas:
event: snapshot
data: {"topPost":[], "user": {"comments": []}}
pending: topPosts,user.comments
event: delta
data: [17,{"comments":[{"body":"hello world"}]},"user"]
pending: topPosts
A good delta format is Mendoza [1] (full disclosure: I work at Sanity where we developed this), which has Go and JS/TypeScript [2] implementations. It expresses diffs and patches as very compact operations.
Another way is to use binary digging. For example, zstd has some nifty built-in support for diffing where you can use the previous version as a dictionary and then produce a diff that can be applied to that version, although we found Mendoza to often be as small as zstd. This approach also requires treating the JSON as bytes and keeping the previous binary snapshot in memory for the next delta, whereas a Mendoza patch can be applied to a JavaScript value, so you only need the deserialized data.
This scheme would force you to compare the new version for what's changed rather than plug in exactly what's changed, but I believe React already needs to do that? Also, I suppose the Mendoza applier could be extended to return a list of keys that were affected by a patch application.
[1] https://github.com/sanity-io/mendoza
[2] https://github.com/sanity-io/mendoza-js