In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem.

I got interested in the Scala as a language and decided to pursue PhD in computer science focusing on compilers. Long story short, I am finishing my PhD on Monday :)

Don't be negative about technology. Be positive.

> In my previous company we used Scala + Spark and I hated it initially. I had no idea how syntax worked, no clue about functional programming or the ecosystem.

It's funny that if you tell Cat-purist that Spark is functional-style, they would laugh at your face :)

That's an unhelpful thing to say. Most people agree that the Spark scala DSL is functional style. It's not as advanced in the style as other scala ecosystems but it has been a very good on ramp for people discovering functional style and is widely celebrated for that

Scala + Spark is absolutely the sane face of Scala. Try using stuff like cats (basically Haskel ported to Scala) to experience the deep end of the pool.

To discard Cats as 'basically Haskell ported to Scala' is doing it a bit of a disservice. Yes, it explores pure functional programming, but that design space is huge.

> Spec2 is a terrible dsl that offers a shitty vocabular to write unit tests and makes the common path incredible difficult compared to junit.

Yes it is. Deprecate using it and migrate away from it. I hate that these shitty overengineered Cucumber wannabees are what "Scala" means to most people.

It's a great language IMO, but you need to make sure fancy frameworks pay their weight. IMO Shapeless and Scalaz-or-equivalent do - replacing reflection or AOP with plain old compiled code, even complex code, is a huge win. But SBT and the mess of Scala testing libraries are just huge complexity for marginal benefit.

As a counterpoint, I used scala at my previous job and it was the best coding experience and environment I ever experienced.

Got rid of 1000s of lines of Java and SQL for 100s of lines of scala, scalding and Spark.

The codebase size on the first migration was 10% of the original Java+SQL. overall, a team of 6 worked for 4 years to produce 10K lines of high quality Scala code for a very complex app and platform after the first transition from 10K lines of java+sql to 1K lines of scala.

Never experienced something like this before or after.

At the next place I wrote 400 lines of C++ mapreduce over 3 days, which could have been 40 lines of sql or 10 lines of scalding/spark.

Jarring transition.

Regarding the reduction in lines of SQL. Doesn't this imply a tradeoff with where compute is utilized? By reducing SQL, wouldn't you be "pushing up" the compute from the database to the JVM while also increasing the amount of data that has to be moved around?

Edit: Wait, did you imply that the SQL you replaced with Scala ultimately gets transpiled into SQL (spark or other dataframe engine) and "pushed down" to the database? Now you're just maintaining Scala and not a rats nest of SQL strings concatenated together? If so, that's pretty awesome.

No. There was no transpiling to SQL. This was several years ago, before Spark SQL.

The SQL was HiveQL. Spark didn't really support SQL at the time and was native JVM Scala RDDs.

HiveQL with UDFs compiled to JVM bytecode typically ended up performing worse than scalding that corresponds to straight raw jvm map reduce jobs that are the real native interface of Hadoop.

The real performance drag from HQL came from developers writing inefficient map reduce jobs with extra joins to avoid writing Java UDFs because they were a huge PITA.

In scalding, developers wrote map reduce jobs with as few group bys as possible. Our hadoop infra team actually complimented us on how great the execution profile of our teams jobs were.

Most of this was Hadoop. Which executes mapreduce jobs written in sql on the jvm. The scalding jobs executed faster, except for one job for which we had to stick with Apache pig.

Spark jobs were native RDDs before the SQL Spark era.

Spark is really its own world that’s fairly separate from the Scala community at large. What makes it a ”terrible technology”?

Scala moved the needle five years ago and made java better among other languages. Using scala helped me grow as an engineer. in my view, it has no place in an enterprise. I require a mature environment that favors consistency. It boils down to reducing complexity. I can barely handle the complexity growth due to the demands of the business, whether it be software engineering or people and these reasons do not justify scalas pluses. Ymmv.

I think scala can absolutely account for that complexity, like it is not a one-way street. It can scale both up and down on expressivity. But I agree that often times it is not used that way.

I wouldn't say spark is terrible, but it is a sort of a "reverse bicycle". If you put a spark codebase down for a week or two and pick it back up, it feels like you're learning everything all over again.

newer java has many similar features that scala had, so it seems moot to use it now.

unless you like ridiculously long compilation times and having libs tied to what version of the compiler you're using (serious WTF)

> it seems moot to use it now

It's true that some features of Scala have made it to Java, but I am not sure the gap between the two languages has shrunk much. For example while Java has added first-class functions and closures, it is still clumsy to attempt any serious functional programming approach with Java; Scala's type system is still way ahead; Scala compile to the JVM, JavaScript, and now native via LLVM.

> having libs tied to what version of the compiler you're using

It's never been that. However libraries used to be tied to the major version of Scala (for example 2.11, 2.12, 2.13) as major versions were not binary-compatible with each other.

Scala 3 addresses this thanks to the TASTY format which creates a strong foundation for binary compatibility. [1] You can use Scala 3 libraries with Scala 2.13 and, more importantly, Scala 2.13 libraries with Scala 3.

In short, Scala is not standing still.

[1] https://docs.scala-lang.org/scala3/guides/tasty-overview.htm...

Well, you know, you can just use Junit with Scala. That's what I do.

It saddens me to see that the HN crowd had been diluted to such an extent in terms of engineering quality that straight up luddism has become the norm, often accompanied with silly ad hominems.

I don't agree with him, but that's a biiit smug..