If we were to stabilise on a framework, I don’t think Java/spark is the right one. It’s notoriously fickle fragile to run, it’s obscenely resource heavy, often not actually that fast, and it’s construction is locked into a set of implementations that have aged poorly-you basically need a JVM wizard to tune everything correctly for you, and you need to run so much supporting infrastructure that I think it greatly erodes the benefits of spark.

> "the new stuff" which could either all be on the same platform or much more likely spread across a number of platforms.

Yes. I’d like to see more variety in approaches, more variety in specialisations, and for interop between “platforms” to occur at a slightly “higher” level (I.e. maybe common sql dialect, common set of messaging/interop protocols).

I think rather than pouring all of our effort into a singular platform, we should put our effort towards improving our tooling and languages so that it is more straightforward and viable to build these platforms.

Everyone language has got their own web-server framework(s), I see Spark et al as the web-server framework of the data space.

I feel like you're conflating Spark and the JVM.

I get your point on tuning Spark, been there, suffered the frustration where a job fails 4 hours in, but yeah, that's Spark for you.