I am a consultant working on Hadoop installations across the globe. As an average I usually able to save 70% disk usage and 30% overall cost by changing the defaults to a reasonable value as well as migrating companies out of HDFS to something like S3. I have spent majority of my career (10 years) working on Hadoop and I can tell you that it is a terrible piece of software with insane ineffciency all over the place. If you would switch over all the Hadoop installations on Earth at once to something more reasonable it would be visible on the global CO2 production chart quite a bit. What is good for the bottom 90% of companies is a financial question not a technological question. It is a bit unfurtunate that Hadoop is so popular and nobody cares about efficiency, not even the Hadoop vendors (maybe with the exception of MapR, which is not opensource).

> This find | xargs mawk | mawk pipeline gets us down to a runtime of about 12 seconds, or about 270MB/sec, which is around 235 times faster than the Hadoop implementation.

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

Using hadoop/spark for <2gb of data seems like a terrible idea.

When all you have is a hammer everything starts to look like a nail.

Well this is great until you need more nodes. :) I am talking about the same scalability while maintaining a much lower ecological and financial footprint.

Is there a good open source alternative that meets the HDFS use-case (i.e. file or blob storage, rather than a KV store designed for point lookups)? Or is tuning the HDFS defaults the best you can do without migrating onto someone's cloud platform?

I'd argue Kubernetes isn't the best choice for the bottom 90%. There's a lot of companies you could describe as tech and many are doing just fine in the old world of manual application provisioning.

I don't agree at all. While the large majority does not need the scalability it offers, it can benefit from all the other stuff applying the 'best practices' offers. The problem is - that many people do not stick to the best practices and do not know how to build containerised applications.

Implementing the whole "DevOps" idea just becomes a whole lot easier when developers don't even have the concept of their snowflake server anymore. And yes - k8s has a ton of overhead and is pretty complex to get into at first, but it all makes sense. There are many points that could be criticised about it, it's far from perfect, but having a standardised way of deploying whatever application has been a massive game changer in the development environments I've been thrown in.

Source/disclaimer: I'm a consultant that has seen quite a few k8s/openshift fuckups and success stories, both on large and small scale.

>I'd argue Kubernetes isn't the best choice for the bottom 90%.

Exactly. Instead of having something so simple that scales for 90% of everyone's needs. We have solution that Most enterprise wants and filter down from top to bottom. And it is true in almost all Tech things related.