Re:Why HTTP/3 is eating the world
by tsimionescu on Thursday October 05, 2023 @09:05AM
(#37778090)
Depending on the source of the data, that is not as good as an actual streaming implementation. That is, if the data is coming from a network API, waiting for it to be "in memory" before processing it still means that you have to store the whole stream on the local machine before you even start. Even if we assume that you are storing it on disk and mmapping it into memory that's still not a good idea for many use cases.
Not to mention, if the code is not explicitly designed to work with a streaming approach, even for local data, might mean that early steps accidentally end up touching the whole data (e.g. they look for a closing } in something like a 10GB json document) in unexpected places, costing orders of magnitude more than they should.
Re:Why HTTP/3 is eating the world
by Thiez on Friday October 06, 2023 @12:58PM
(#37793208)
JSON discourages but does not forbid duplicate keys. In case of duplicate keys browsers generally let the last instance win. So if you want to be compatible with that, you always must read the whole document.
Not to mention, if the code is not explicitly designed to work with a streaming approach, even for local data, might mean that early steps accidentally end up touching the whole data (e.g. they look for a closing } in something like a 10GB json document) in unexpected places, costing orders of magnitude more than they should.