Re:Zstandard – Real-time data compression algorithm
by BackBlast on Tuesday September 06, 2022 @12:04PM
(#32739031)
I use brotli as a web distribution format and as an internal compression format on web stacks for various projects and for production systems without issue.
I still don't understand the use case limitation, or exactly where this is a potential issue. Why is the data being interrupted? Is this some kind of raw UDP stream use case?
Common reason being someone forgot to check how many bytes read() actually returned and just assume it filled the buffer.
Robust data formats have double protection against this class of bugs with unambiguous SOF and EOF boundaries that the consumer can assert. I guess a sanity-byte once every X Megabytes wouldn’t hurt either.