News for hackers. Stuff that matters.
Re:Ask HN: How to return large response over an API? by tiew9Vii on Saturday May 08, 2021 @04:35AM (#27085128)
If you can chunk the response json-seq/xml (sax parser) may be worth looking at.

The server can incrementally stream a chunk and client incrementally consume a chunk keeping flat memory usage.

I think gRpc also supports streaming but don’t know much on it.

JSON is a bad format for large files as generally you need to read the entire file in to memory before you can use it as you observed.

Re:Ask HN: How to return large response over an API? by cloverr20 on Saturday May 08, 2021 @04:48AM (#27085185)
This is the exact problem I have been referring to. Parsing json without loading it entirely is difficult, will look into the xml (sax parser). Can the yaml format be considered for a streaming response approach?