The thing that has killed Hadoop, imo, is Kerberos. In Hops, we have switched to using TLS/SSL certificates instead of Kerberos, and that enables us to implement dynamic roles. Dynamic roles allows us to build a software-as-a-service platform, where projects are securely isolated from one another.
I agree Hadoop is no longer MapReduce. It's HDFS+YARN. That's it. Distributions package up Spark/Flink/Kafka/PrestoDB with the HDFS/YARN core.
At Hops, we've scaled the core HDFS by >16-37X ( https://blog.acolyer.org/2017/03/06/hopfs-scaling-hierarchic...) and we have a distribution called Hopsworks with support for Spark/Flink/Tensorflow. Nobody uses MapReduce on our platform.
The thing that has killed Hadoop, imo, is Kerberos. In Hops, we have switched to using TLS/SSL certificates instead of Kerberos, and that enables us to implement dynamic roles. Dynamic roles allows us to build a software-as-a-service platform, where projects are securely isolated from one another.