News for hackers. Stuff that matters.
Re:Binance built a 100PB log service with Quickwit by francoismassot on Thursday July 11, 2024 @10:18AM (#40937096)
Indeed. They benefit from a discount, but we don't know the discount figure.

To further reduce the storage costs, you can use S3 Storage Classes or cheaper object storage like Alibaba for longer retention. Quickwit does not handle that, so you need to handle this yourself, though.

Re:Binance built a 100PB log service with Quickwit by AJSDfljff on Thursday July 11, 2024 @10:26AM (#40937173)
I would probably build my own storage pods, keep a day or a week on cloud and move everything over every night.
Re:Binance built a 100PB log service with Quickwit by jcgrillo on Thursday July 11, 2024 @12:40PM (#40938374)
Logs should compress better than that, though, right? 5:1 compression is only about half as good as you'd expect even naive gzipped json to achieve, and even that is an order of magnitude worse than the state of the art for logs[1]. What's the story there?

[1] https://news.ycombinator.com/item?id=40938112