One great side effect of this was service developers weren't afraid to write logs. We logged excessively, and it didn't cost too much. If we'd been indexing everything in ES it would have bankrupted us.
These days with S3 and the cloud, hadoop (or the EMR suite) per se probably isn't the way to go, but I'd sure like to see observability solutions giving me a first-class programming model that I as a user can interact with--not some bespoke "query DSL", and for them to accept that instantaneous indexed retrieval isn't important.
This paper is really interesting: https://www.usenix.org/system/files/osdi21-rodrigues.pdf
Stuff like this gives me hope we can have it both ways. With highly tuned compression and programmatic access the user is empowered and the cost is minimized.
https://www.youtube.com/watch?v=XXgBJmqv0ok