News for hackers. Stuff that matters.
Re:AWS switch from gzip to zstd – about 30% reduction in com... by Someone on Saturday August 20, 2022 @07:34AM (#32531209)
If I were Amazon, researching whether zstd can be used reliably could take quite some time. It would require a thorough code review, inspection of the assembly produced by compilers, and a test set of a few terabytes.

More importantly, it would require researching whether something even better might be just around the corner. I don’t think Amazon wants to switch compression every few months.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by unboxingelf on Saturday August 20, 2022 @11:14AM (#32532613)
Huh. If an IC had a compelling argument for cost savings like this, I’d recommend they pursue the idea as a 10% style project. First verify the basics: licensing, health of the community, etc. Then build a standalone pipeline using the new protocol. Process data, collect benchmarks and metrics. Present your findings to the team and manager. If the manager thinks it’s still viable, they go up the chain and involve principals, architects and so on.

In summary, lower the bar for trying new things. This is how we innovate.

Re:AWS switch from gzip to zstd – about 30% reduction in com... by antisthenes on Saturday August 20, 2022 @05:34PM (#32535310)
That's literally what the person above you said. On Amazon's scale the process will take time.