> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case

Scala the language is quite fine, but it attracts this kind of programmers. The damage they do to a codebase has to be undone by someone who both understands those advanced concepts and is not a FP fanatic.

> Scala the language is quite fine, but it attracts this kind of programmers.

It does. I call them "code scientists": engineers want to learn a language to build things, scientists want to build things to learn the language. I've interviewed >100 Scala engineers and if the panel thought they were the kind of person that just wanted to understand FP instead of solve problems, they were rejected.

I am now retired, but Scala is still my goto choice for engineering software. It checks all the boxes on concurrency, bulk I/O, ecosystem. In the hands of a sane and pragmatic practitioner, it's hugely productive.

I read this and I get so tempted but then I load up the homepage and see "JVM" and regain my senses. You almost got me.

Well, Scala compiles to JavaScript and native through Scala.js and Scala Native. Plus you can also compile to native with GraalVM Native which works very well.

Scala compilation + graalvm compilation = very long breaks :)

Interesting, in my experience long compile times in Scala equates with a bad code base. And invariably involves an uber object with a whole lot of implicit conversations and is imported on every other source file.

I think the pertinent part of the comment you're replying to was graalvm, not so much Scala. I was compiling a small Scala codebase that took 15 minutes to compile - most of that was not my code, it analyses the reachability of the entire classpath and compiles it to native machine code.

I don’t know about bad code but a lot of derivations that send compiler into proof loops usually is what is causing slowdowns. You can write the same code by hand, pick your poison. Honestly, scalac is probably still faster than typescript compiler and somehow no one complaining about ts

> I call them "code scientists": engineers want to learn a language to build things, scientists want to build things to learn the language.

I think this is a bit contradictory as it is phrased. Learning the language is essential to use it effectively. Otherwise one is better off with a language already known. And it is very possible to build things to learn the language and still keep it sane.

If we went back to let's say 2014 "when the cake pattern was the solution of all problems in world" I think I'd agree with your statement, even back then when I was a bit of a Scala fanatic.

> who both understands those advanced concepts and is not a FP fanatic.

Oxymoron.

I speak monads. I am a 50 yrs old high school dropout. Cool thing is - just because you are not seeing FP structures in non FP focused languages doesn't mean they aren't there.

Oxymonad?

I worked at Databricks in the past and Scala is just a way to get things done there. Honestly, for me writing code day-to-day wasn't that different from Java once you understand how to structure your classes. Handling nullables with typed Options is pretty nice though, and I miss it in other languages now. Most people who join Databricks do not know Scala beforehand, it's just a useful tool.

Sure, sometimes progress is slower than you'd like, but that's the case in every large monorepo.

> Some of the prior employees enjoyed playing with advanced language features and writing libraries for the most general possible case even when that made it hard to understand how they were used for the 2 actual cases we needed to handle. Akka, Cats, and Shapeless were all over the place.

This hurts me a lot

I considered it a near miss when a Scala shop rejected me because I used Maven to compile their coding challenge. Clearly the company was full of over-complicaters and actively recruiting for it.

How did that conversation go?

“We like your solution but did you seriously just use Maven?”

What did they say exactly?

Yeah that’s got to be up there on the list for dumbest technical related rejection I’ve ever heard of, particularly if the code was done well otherwise.

As opposed to Gradle or SBT?

If I'm being honest here, a quite significant part of day-to-day backend JVM software development positions is dealing with the build tool. While I wouldn't outright reject a candidate for not being familiar with the latest standard toolchain (Gradle), I would certainly consider it as one of many factors - Gradle is pretty complicated, yes, but also quite powerful in what it is capable of achieving. It's not as eyebrow-raising as, say, if you sent in a coding challenge compiled using Ant (which would, in fact, be quite a fun discussion to have with a candidate) but it would certainly be something considered.

> If I'm being honest here, a quite significant part of day-to-day backend JVM software development positions is dealing with the build tool.

perhaps, because your work has over complicated it [1].

[1] If you are doing multi-distributions or multi-repo stuff perhaps it's warranted, but those complex build features better be providing serious value to the business to be worth losing 10% of the development workforce's time on build tooling.

I also prefer to use Maven to compile Scala code - it's much faster, better integrated with IDEs, etc.

That's silly but unfortunately this type of silliness is the norm in the industry.

While easy to understand, I’m constantly surprised to see this type anti-intellectualism. The fact that you don’t have the required prerequisites to understand a codebase doesn’t mean it’s bad. Educate yourself on Category Theory and functional programming techniques and learn to leverage these tools to your advantage.

Alternatively you can work in Go where braindead simplicity is the mandated norm.

Thank you for this reply. It demonstrates exactly the mindset of those few FP die-hards that I have seen wrecking havoc in a couple of companies where I worked for.

I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were those that spread the evangelism of design patterns to all places regardless if it made any sense; before that in the 90's there was a group of programmers that liked to generate code until no colleague understood anymore what was happening. And when you tried to tell them that abstractions don't come for free, but with a cost because they make code harder to understand, the answer for the last 25 years has always be the same, although rarely said straight: that it was only hard for -you- to understand, and not for the enlightened master himself.

I have seen more projects fail because of too much unneeded abstraction than by all other causes together. I even have seen companies go almost bankrupt because of projects engineered by lone wolfs where nobody understood the abstractions anymore except for the designer himself, and at some point he himself not really anymore either.

And the problem is that this mindset continues to be cultivated by CS books and conferences. Few people want to read a book that tells them that the secret of being productive as a team depends on mostly the culture in the team and the simplicity of the code, and not on the latest hyped framework, language or paradigm.

> I'm not saying FP is bad, it is actually really powerful when it is in the hands of those who understand when and how to use it. The problem with the attitude that you demonstrate in your reply is also not limited to FP: in the era of OOO, there were those that spread the evangelism of design patterns to all places regardless if it made any sense; before that in the 90's there was a group of programmers that liked to generate code until no colleague understood anymore what was happening.

Agreed, and what did we get? dogmatic decrying of how OOP is completely useless and objectively bad, not too dissimilar from some of the comments on FP and Scala on here.

> dogmatic decrying of how OOP is completely useless and objectively bad

And just who were the ones pushing this? The people peddling FP.

The people in this thread are not pushing anything. They're just sharing their experiences with Scala and how unproductive it is to deal with FP zealots.

I found this in another thread and it sounds like you might enjoy it also:

https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction

> before that in the 90's there was a group of programmers that liked to generate code until no colleague understood anymore what was happening.

Ah! Just write a quine and be done with it!

> I have seen more projects fail because of too much unneeded abstraction than by all other causes together. I even have seen companies go almost bankrupt because of projects engineered by lone wolfs where nobody understood the abstractions anymore except for the designer himself, and at some point he himself not really anymore either.

"cake pattern" on one side and scalatest on the other.

As for

> I’m constantly surprised to see this type anti-intellectualism.

You hardly need any Category Theory to theory to understand type theory or mostly important Scala's type system. This the exact knowledge and allows you use it effectively. Yeah, the way he puts it makes those who like to understand the theory behind everything look bad.

For most tasks programming it's more an engineering discipline than an intellectual pursuit. In that context complexity must always be justified.

I know, and have been told, to have written hard to read code. In my case it's usually vectorized code in numpy or a C or cython extension for really hot code. But I always have a good reason, usually performance when it matters.

It's usually said that premature optimization is the root of all evil, but it's nothing compared against premature abstraction. Optimization at least is local in its nature while abstraction tends to expand all over a code base, and when the assumptions made for the abstraction no longer hold people is still forced to keep dancing for a music that no longer plays.

"Educate yourself" assumes that your interests are everyone's interests but this is a vast field and not everyone has interest nor time to learn about category theory. Most probably your coworkers are quite intelligent. If they are not interested in your idea it may be that it's not appropriate or maybe not correctly framed, or not mature.

I'm stealing that phrase: "Premature abstraction is the root of all evil". It's hard for me sometimes to justify in a code review why an abstraction is not required (yet) when someone has put some effort into it.

I found this in another thread and it sounds like you might enjoy it also:

https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction

The thing is I DO understand category theory and FP after going off the deep end in Scala land. It did not make me a better programmer or make writing code any easier or faster. In fact it makes me spend 50% of my brain power on juggling those concepts and how they are encoded into a language that does not and never will support them as well as say Haskell.

I learnt the basics of FP a long time ago and understood the data-centric view of computation, the benefits of immutability etc - those things actually do help me write better code. The advanced FP, not so much. The nightmarish encoding of advanced FP shoe-horned into a language not built to support it that compiles into Java definitely not built to support it actively hurts. There are so many dirty macro shenanigans and hacks in the guts of the FP-purist libraries to force the language to implement things it just isn't particularly well suited to implement, which makes reading or debugging library code a nightmare. Like if you want Haskell, just use Haskell FFS and let us code reasonable Scala in peace.

There are so many parts of math that are much more useful in various fields - basic optimization in solving sudoku puzzles[1], linear programming in Z3, gradient descent in ML, calculus & vectors in graphics programming etc.

[1] https://norvig.com/sudoku.html

Everytime the “go learn category theory if you want to become a better programmer” trope arises on HN (less often than it used to, but still occasionally) I’m left scratching my head. I actually learned category theory as a math grad student, before changing fields and going in a more applied direction. I’ve spent decades writing software since then, and I can easily think of about 10 areas of math and CS that are way more useful day to day. Once you’ve mastered calculus, linear algebra, probability theory, statistics, numerical analysis, optimization, algorithms, complexity theory, combinatorics, asymptotic analysis—sure, go pick up category theory.

But Category Theory is different to all the other examples you gave, as it describes program composition. i.e. building larger programs from smaller ones. Computer scientists have been searching for the "mathematics of program construction" for a long time, desperately trying to make the process more formal. Category Theory is what many seem to have arrived at. There are even books about it, e.g. "The Algebra of Programming".

Most computer science graduates are well schooled in algorithms, complexity, linear algebra etc. However, the vast majority of their time commercially will not be spent implementing some core algorithm, but instead gluing systems together and consuming/writing APIs. Any mathematics that helps them better accomplish this would be good to add to their toolbox.

The author of "The Science of Functional Programming" book (https://github.com/winitzki/sofp) makes an argument that CT is at best tangential to practicing FP.

I found category theory to be helpful in giving a vocabulary for some very simple and essential things that otherwise get lost. My goto example are Monoids and Semi-Groups, which are almost mindbogglingly simple but pop up everywhere.

Category theory gives you the precise language to talk about such stuff and helps to communicate and talk about more complex concepts and that's always a plus.

I hear you. However the amount of category theory you need to understand 99.99% of FP is really really minimal. 10-15 pages of well written text

Y'know what, I'll bite. I can do 10-15 pages. You have a link?

this one is not bad http://www.mas.ncl.ac.uk/~ndjw1/docs/djw-ctfp.pdf

Thanks. I appreciate the link. I'll take a look after work.

I sympathize with your thinking. There are two "readabilities".

1. How easy it is for an average programmer to understand the code.

2. How understandable the code actually is if the reader is fluent with that prior knowledge.

If one finds the latter is much better than code written in conventional techniques, but people are scared off by the former, it can get pretty frustrating, and "people are stupid" may be a conclusion drawn.

---

But my experience is that those techniques do not yield better code.

And "Educate yourself" made me barf.

So, I’ll probably get shit on for this, but I really cannot wait to leave this industry.

I hate how “best tool for the job” really just means “what we can have a large hiring pool of people who’ve only ever learned one programming style and possibly only one language of” or “for legacy reasons this is the only choice”.

Yes I’m familiar with reality, I just hate it.

Yes, it's disappointing but hardly surprising to see the phrasing of management parroted by so many developers. It's their job to worry about hiring, deadlines, etc. It's yours to make sure things don't shit the bed at one in the morning, especially if you're going to be paged about it. There are some languages that make it very easy for grads to deliver features, and others that are far more likely to be correct at compile time. The "best tool" is born from these competing demands; those too meek to engage with this inherently adversarial process don't do themselves or anybody else any favors.

Apparently "best tool for the job" means find he bottom of the barrel that can still engage somehow with the codebase. This Idiocracy went to the point where returning function pointers gets questioned during code review because "this might be too complex compared to imperative call".

The code base had multiple problems, none of which I would blame on category theory or Cats.

- Engineers had written higher abstractions seemingly just because they could. When I audited how internal libraries were used across our services, calling applications weren't making use of the advanced abstractions. I'm talking about things like using Cats to abstract across AWS S3 error handling. Cool, except that it wasn't needed because we never actually encountered the exotic compositions of failures anticipated by the libraries.

- The abstractions written for our own business logic were worse than abstracting over S3. They were premature. Our business logic had to change frequently because the end user experience was still evolving rapidly. Changes that violated previous assumptions and their corresponding abstractions took longer than they should have and/or led to very awkward code.

- At least at the time, tooling had more problems with the "advanced" code. The IntelliJ IDEA Scala plugin could not yet show how implicits were used. It couldn't find senders sending to an Actor the way it can easily find plain callers of a function. You would need to manually force a "clean" in certain modules before code changes would compile as expected. IDEs would also fail to flag code that couldn't compile, and incorrectly flag code that would compile, at a higher rate compared to plainer Scala.

I'm still glad that I have access to Cats, Akka, and other advanced parts of the Scala ecosystem. They're still used in a few places where their value is greater than their cost. Even in the plain code, I'm still very glad I have pattern matching, immutability-by-default, rich collections, map, flatMap, filter, fold, scan, find, etc. I have no plans to transition our company off Scala internally. If I were starting a greenfield project with myself as the sole developer, I'd probably be using Scala for that too. But I prefer to write a bunch of simple repetitive code first, then develop abstractions after it's clear what the commonalities are.

There are others options besides Go and category theory. I'd say that for now, languages that are like ML are enough. They're coming in the mainstream with pattern matching, immutable records, etc. It'll take some time for people to learn this, learn to use it properly, teach it to others. And maybe some years later a monad will be a pattern as common and well-known as an iterator. But today isn't that day yet, and until now you have to collaborate with your peers. You also have to assume that you may be wrong about some of this stuff, and keep an open mind for FP alternatives. For example, I think algebraic effects aren't part of category theory, yet they're an exciting new feature that might have an impact in "industrial" languages one day.

The future is unevenly distributed.

There are plenty of companies where everyone has a very mature understanding of monads and where new joiners are coached to reach that level of understanding.

And there are plenty of places that aren't using source control, CI and unit tests.

If all you do is add numbers together, is it anti-intellectual to question why one needs to grok Principia Mathematica first rather than just doing some arithmetic?

In other words, do you solve problems that actually require category theory or are you just navel gazing?

If you've ever done a map reduce on a dataset you've used category theory, whether knowingly or otherwise.

You don't need more than highschool maths to understand the category theory behind common typeclasses.

This is development nowadays, hire people fresh out if am eigh week javascript boot camp and everyone has to write code they can understand.

Instead of bringing the juniors up, we drag the more experienced guys down to their level.

The result is really shit basic code everywhere. or you end up using a language like go where there's basically only for, if and arrays and writing code in it is miserable and tedious.

This ain't anti-intellectualism. Scala fanatics (more like FP fanatics who use scala) write overly completed code just for purity's sake. Its ok, but it creates a maintenance nightmare and a huge hiring problem.