Re:Ask HN: How to return large response over an API?
by cloverr20 on Saturday May 08, 2021 @04:48AM
(#27085185)
This is the exact problem I have been referring to. Parsing json without loading it entirely is difficult, will look into the xml (sax parser). Can the yaml format be considered for a streaming response approach?
The server can incrementally stream a chunk and client incrementally consume a chunk keeping flat memory usage.
I think gRpc also supports streaming but don’t know much on it.
JSON is a bad format for large files as generally you need to read the entire file in to memory before you can use it as you observed.