If I were Amazon, researching whether zstd can be used reliably could take quite some time. It would require a thorough code review, inspection of the assembly produced by compilers, and a test set of a few terabytes.

More importantly, it would require researching whether something even better might be just around the corner. I don’t think Amazon wants to switch compression every few months.

Huh. If an IC had a compelling argument for cost savings like this, I’d recommend they pursue the idea as a 10% style project. First verify the basics: licensing, health of the community, etc. Then build a standalone pipeline using the new protocol. Process data, collect benchmarks and metrics. Present your findings to the team and manager. If the manager thinks it’s still viable, they go up the chain and involve principals, architects and so on.

In summary, lower the bar for trying new things. This is how we innovate.

That's literally what the person above you said. On Amazon's scale the process will take time.

He did not say in the tweet zstd was implemented at AWS in 2022, could have been implemented years ago

When I was there last over a year ago, there was a big push to switch logs stored in the internal log service from gzip to zstd. But gzip is extremely well integrated into essentially everything, that it's a little annoying to switch. For example, the base OS images didn't even have zstd installed. It's still worth it for the benefits (at least once all the internal tools started supporting it), but it's not without some downsides. Which is maybe why it took so long to happen in earnest.

Gzip was written extremely well. I have been trying to get static compression with Brotli working for compressing web assets, along the way of getting that worked out, you realise that few people can be bothered with it, hence you are having to compile your own Nginx modules.

I found out the way to do it was to have Nginx run a proxy server that caches the output of another Nginx 'server' that does the Brotli dialled up to 11. So you are making your own CDN.

Nothing is difficult when you know it, but Gzip has scratched the compression itch so well that people just do not have a problem with it and therefore do not seek to change it.

There is also the cost of a bug. Imagine that AWS corrupted customers data due to a compression bug in your home-grown compression code.

It is almost like encryption in this regard.

Luckily by adding a checksum of pre-compression data it's very easy to catch bugs before they do widespread silent damage.

Amazon may be big but even then resources are limited. Individual teams owning the roadmap for services are still small. And there are always competing priorities. Do you prioritize these security and availability improvements or do you pickup some operational improvements to make our engineers oncall easier? Oh there is also this long line of customer features you need to deliver. And re:Invent is also approaching.

When you look at it like this, not very surprising that initiatives like cost savings optimizations may take a back seat for periods of time.

> I want to know why a company as big as Amazon was still using gzip in 2022....

The short answer to "Why still using gzip in 2022" is almost always answered by "Because it made no business or financial sense to spend head count on it".

Amazon is usually fairly smart about the ways it spends head count, particularly on things that could notably reduce costs. Managers/directors/VPs/SVPs obsess over what the value is from various work, and routinely adjust priorities and reallocate head count. They track not just what is happening within individual services, but also the overall business strategies, what's on the horizon etc.

One simple example that most folks outside the industry won't be seeing is that every major cloud has been working their collective arses off for well over a year on meeting JWCC contract needs. There's a lot of work involved in that contract, because unsurprisingly it's really hard to build and run an entirely air-gapped cloud region, but the payoff on being selected as a vendor is phenomenal. Far more than saving an additional 30% of storage, even at AWS S3 scale. _It's not the only major business and engineering initiative_ that will be taking place across AWS.

I've got a long list of "We should do x, y, z" that applies to my service in the cloud I work for, a number of which will see notable performance improvements. They make zero sense allocating head count to, though, because I've got at least a dozen other more important things going on from a business roadmap perspective to get solved that will make us way more money and/or save way more engineering resources down the line, be it automation stuff, or new features.

What does happen, though, is that list is kept in mind any time new business priorities come about. If there is any way bits on the list can be tied in to something the business wants, it'll get hooked in to it. The business gets what it wants, the service gets what I want, everyone is happy.

(To repeat something others have pointed out, there's also zero indication of timeline, it could have happened any time in the last several years that zstd has been a thing, they may well have switched to it almost as soon as Facebook removed that ridiculous "You can't sue us" clause)

How much does it cost to switch? How much effort are you putting into making sure there are no pathological cases while encoding and decoding that have an outsized effect on resource use? How much risk are you willing to take that you don't lose any customer data? If you recompress stored data, how much are you willing to pay in compute? Recompressing at AWS scale is enough IO to burn through many thousands of disks in aggregate—how much does all that wear cost? How much tooling do you need to rewrite?

Let's say you save a few hundred million dollars a year after switching. The cost to switch is almost certainly more than a few hundred million dollars. When you make that kind of investment—especially when you're moving away from a very boring technology—you want to be damn sure you know exactly what you're getting yourself into.

The tweet doesnt mention dates or the coverage.

Second I believe youre underestimating the scope of the problem. Amazon has thousands of teams, even more services, each with their own priorities, and innumerable different access patterns, data types, etc. For all practical purposes there is no single “they.”

To get an idea of the scope how long would it take you to remove gz from oh … 50,000 projects? Each with many deployments and up to 15 years of active data.

Because finishing the next product you can sell can produce more revenue than the savings from switching to zstd.

The beauty of gzip is that it is effective, free, resource efficient, and it works with everything.

A parallel implementation like pigz is still pretty good and widely backwards compatible.

I'm surprised they aren't using an FPGA solution

Knowing such implementation exists, is open source, and since they don't state what they're using... you may just be right

https://pure.tudelft.nl/ws/portalfiles/portal/94907930/Jiany...

FPGAs are really expensive in volume compared to other options, so it's not surprising to me.

But on an FPGA you can probably implement super fast and effective compression.

If it is far enough above the curve of speed Vs ratio, then it will pay for itself in saved storage.

Remember there is no need for said FPGA to be in every machine - in a data center with ten's of gigabits of bandwidth to every node, you can send data to another machine for compression and receive back the compressed data to store.