There's a blooming ecosystem of pure functional programming libraries. Or even two ecosystems: Typelevel and ZIO.

My company uses Scala a lot with with these ecosystems (my team is on Typelevel, another one is on ZIO) and in fact Spark and Akka is what we're trying to avoid as much as possible. Once I got used to the composable nature of FP and static types - both Spark and Akka just feel clumsy and unintuitive, like maintainers of Spark chose Scala because it was a "Java with fancy syntax" back then, ignored most of benefits of static types, HKT, immutability, type classes etc. If there's future behind Scala - it definitely doesn't lie in Apache land. Akka is slightly different and probably has higher potential, but just an overkill for most of use cases it's advertised for.

def mirrors my experience too. Vast majority of spark jobs were easily ported to sql/dbt and the remaining ones are in pyspark. I used to use a lot of scala spark in backend data processing in 2016 but now its almost down to zero.

scala is real big impediment to making data processing accessible to general public in your company. order of preference now at my company is,

1. sql 2. pyspark 3. java spark 4. scala spark

eg: shopify found that 70% of their pyspark could be converted to just sql https://shopify.engineering/build-production-grade-workflow-...

I've rolled out Scala based Spark interfaces to non-programmers in Databricks notebooks, so it's definitely possible, but only if you stick with the basic language features.

Here's a more detailed PySpark vs Scala comparison in case folks are interested: https://mungingdata.com/apache-spark/python-pyspark-scala-wh...

I think Scala Spark (using 10% of the language features) is a better technical decision (because it provides huge benefits like fat JARs, shading, better text editor support, etc), but the worse overall choice for most organizations because people are generally terrified of Scala.

They'd rather do nothing than write Scala code. I can empathize with their position.

Even when Scala is used more or less like python?

> scala is real big impediment to making data processing accessible to general public in your company

Ding Ding Ding! Presto/Athena now is becoming huge in BI ecosystem. We don't really use Spark for ad-hoc BI anymore, we use it for data science and large repetitive workload.

I want to like Scala and Akka, but they just seem to get used very little in the “real world”. I recall some HN comments complaining about how a lot of the Scala they were writing for Spark was more or less a series of procedural API calls rather than using the language to its potential.

Yes, totally. We are using Scala for web development, without using spark. Half of our services run on akka, the other half uses typelevel ecosystem.

It's used in my workplace for ElasticSearch interactions. It's for search, product recommendations, discovery, sorting in an ecommerce.