There is a tendency I observed in engineers to not want to appear dumb. This is a recipe for disaster when coupled with a deep language like Scala, because nobody wants to solve problems the simple way, and Scala gives a tremendous amount of rope. That's why I think it's only a suitable choice for very experienced or disciplined teams, with a strong consensus on style or any DSLs. I can see average enterprise shops lacking these things and ending up with suboptimal results if Scala is used.

> to not want to appear dumb

Few production incidents are enough to change this mentality. I'd rather write more verbose but "obvious" code than "clever one liners". I feel sorry for anyone who has to understand someone's DSL on the fly when there's an outage and your business is burning through X$ per minute of downtime.

(On the other hand, you probably want to structure your production changes in the way that they are easy to roll back without understanding everything, but that's other conversation. Either way, the thought that someone may want to chase me after work because my code broke and they can't understand it is enough for me to stop being clever).

I think Scala needs to be treated something like how smart shops treat C++: there's so much there you can hang yourself with, so you need to define a subset/dialect and create a style guide and stick with it.

This is how C++ has been successful at Google, and it's how I'd approach something like Scala if I were to go back to doing it now.

I don't disagree, but Java keeps adding features that make it closer to Scala, and Kotlin is well-supported and has cleaner syntax, so you have better choices than C++. The space between C and C++ is too fragmented, and too many of the languages are niche.

Yes, I haven't been in the Java world for almost 10 years now, so I can't comment too much on that, but it seems like Java improvements & Kotlin have vacuumed up the reasoning for using something like Scala. When I used it 12 years ago, the core Java language itself was terribly anemic.

Java stagnated for a long time at 7. Java 8 really started the change and it has been moving forward at a good pace since then.

I'm going to say maybe too fast. Not that the changes aren't good (they are good) but rather that the pace of features from 8 to 11 and then 11 to 17 has been fast and trying to get old toolchains updated to 8 (yes, it pains me) - much less use the features that are present in 11.

While in the organization that I work in there was some "ok, we've got old things on the computers, update to the latest Java now?" was postponed until September so that we can say "Java 17 is the standard" and have that be our long term platform for awhile.

But there are a lot of people who are still writing code that would compile in 7.

Looking at the version history, 12 years ago would have been Java 6... there were a lot of good changes with collections in Java 7. https://www.oracle.com/java/technologies/javase/jdk7-relnote...

The try with resources and catches were the two big things that made 7 much better than 6 for me. And with 8 - https://www.oracle.com/java/technologies/javase/8-whats-new.... - while:

* Classes in the new java.util.stream package provide a Stream API to support functional-style operations on streams of elements.

is only a single bullet point, that's a big thing.

That's how it is successful. HN said my post was too long, but here's my rebuttal to the op and it pretty much goes along with what you said: https://gist.github.com/jackcviers/a7e74a3ad0a57f6ab97afd25c...

If you are in the databricks Spark ecosystem -- stick to using the Databricks style and Spark libraries. Don't bring in anything that isn't in that ecosystem, because it runs way behind standard scala -- it's on a 6 year old version at this point.

If you don't want java to eat the features you use, use either the typelevel ecosystem or the ZIO scala ecosystem of libraries, use their respective preferred built-in DI methods (tagless final and ZIO environments, respectively).

If you don't mind eventually having java eat your features, go ahead and use the Lightbend ecosystem and buy the support license. If you don't want to buy support, the twitter/finagle ecosystem is basically the same without a support possibility. But within those there's definitely room for creative license.

All in all, the functional and typelevel subset works best as it's a tiny footprint of features and libraries, it doesn't leave any gray area for personal expression, and treats the language as its own language rather than clone - you - a - c++ - java - javascript - ruby.

Would I rather everything move to Mill? Yes. Is that ever going to happen, nope.

Some of the twitter ecosystem works fines with typelevel style. Finch, in particular, comes to mind

An evergreen conversation:

"I love C++." —Alice

"Oh? Which C++?" —Bob

Every successful C++ team defines its own C++ subset.

This is why I can't figure out how for the life of me to learn C++ effectively for professional use.

In reality it's not really like that. Yes different shops are going to be using different C++ features in different ways but they usually share some common "core" of the language.

One approach is to get a sample of code from the domain you want to inhabit, preferably including some from the very people you want to work with, and learn the subset of the language they use.

Also Stroustrup thinks that is the way to use cpp, his talks are all about 'do not use all of cpp, pick your features'. Makes senses in a way but it does lead to various notion of what cpp is.

That is exactly it. A company wide style and coding guide is a must with scala

A company wide style doesn't quite get it done. Is your style not to allow high kinded FP coding? (because it is too hard to understand). If so does that mean you don't allow libraries written in that style, like SLICK?

We had PHDs and very senior engineers, and they thought debugging common libraries like slick would take them weeks of work and they opted for avoiding that entire type of scala. But that is hard to do, because its a lot more than a coding style.

I do not think you do much high kinded Fp coding in slick other than "lifting" values from scala code to sql, though I havent worked with slick for in a while. I am not sure why they exposed internal types as we didnt really have any control over them

Perfectly said(!).

I may have a contrarian opinion: a new language, beyond certain level, does not increase productivity tangibly for a large enough team. Specifically, Scala does not necessarily offer more productivity than Java. It is a pleasure to write program in a language with more powerful features, for sure. It's just that bottlenecks of project are usually not language features, but core algorithms, system designs, meticulous testing, conflicting requirements that demand careful trade-offs, availability of robust libraries and frameworks, maintainability of the most complex part of the system, and availability of qualified engineers. Few of such bottlenecks can be removed by switching to a language like Scala.

Can't language features help solve some of those problems though? Improvements in static typing can help with algorithm implementations, reduce the amount of testing required, and make conflicting requirements more obvious.

I only witnessed from the outside, but saw a similar story play out. A client had bet on Scala for a new platform and made it about 6 months in before throwing in the towel and rewriting the whole thing in in Java. It's too hard to recruit and the tooling isn't mature enough.

I didn't run into problems with recruiting in general being too difficult. Plenty of people are interested in FP and are eager to do it professionally. But recruiting the right mix of people can be a challenge. Scala tends to attract people with an experimental temperament, and scare away more conservative developers. A team needs a healthy mix of both, though. It's the creative tension among different attitudes about how to write code that yields the best work in the long run.

Yeah, good luck when someone with experimental temperament decides to implement some of your crucial functionality with semialgebras. (that's what happened in my previous job where we run Scala). I'm happy I didn't have to debug any customer issues around that module. Also, I didn't enjoy constant bickering with my reviewers what makes a beautiful code or not. Apparently there are five ways to do everything in Scala and I always ended up picking not the one my teammates would like the most.

Yes. That's pretty much how the Scala project I was working on ran itself aground. Everyone got so caught up in flexing at each other that solving actual business problems became a secondary priority.

This is why I think the community is way more important than the language. In Python people are like, hmm, this is a hack, but it works for now, in Scala it’s, what have you done you savage?!?

It's also just really weird, when you step back and take a good long look at it, to see people getting worked up about hacky ways of doing things within the context of a language that implemented sum types the way Scala implemented sum types, and that implemented typeclasses the way that Scala implemented typeclasses.

Which, I don't want to be too down on Scala. Overall, I like the language. Scala's real weak point is its culture. There's a decent risk of cognitive dissonance when you try to wrap an ethos of design purity around a language that's always been a bit of a communal experimentation project.

This points to a culture problem more than the language itself though.

Interestingly, when we were working on the Java code, we didn't have this problem. Same people, same time, same git repository, different modules.

This doesn’t terribly surprise me. Community isn’t just the people but also the customs and idioms within the community. It’s probably similar to ‘code switching’ in verbal languages. People talk (and act) differently depending on who thay’re talking to.

Were semialgebras a right abstraction, or over-engineering?

What is helpful in situations like this is the motto "as simple as possible, but not simpler". During a code review, if you see something you think is too complex, ask the author: "How could it be made even simpler? If not, how exactly a simpler approach did not work?" It sometimes helps find an overlooked simplification.

> Were semialgebras a right abstraction, or over-engineering?

I think that's missing the point.

The criterion should be: is the code clear enough to be easily testable, easy to debug, and easy to evolve?

I will take a wrong abstraction with the qualities listed above any day of the week.

This is a good call.

The problem with wrong abstractions, though, is that they do not work, especially when you make the next step on the roadmap.

I would say that clear, testable, evolvable code with little abstraction may be fine. OTOH boilerplate and copy-paste prevents easy or well-controlled evolution. The abstractions end up inlined and fused in the code, instead of being made visible, and become easy to miss a case when a concerted change is needed.

In contrast I think that going all in on Java scares away people who like to experiment.

Be honest and give a short list of experimental approaches that you wanted to do and couldn't be done on Java. What did you want to do and couldn't?

And, let's ask Rich Hickey. What language was it he used to experiment with a new s-expression language? Why wasn't RH scared away? You want the brutal honest answer? Because he is smart and can grok the complexity of the Java in toto.

What sort of experiments can you not do on a virtual machine based language, with open byte code/vm spec, open class loaders, and compile and runtime instrumentation and meta- capabilities?

I used to do the switch the superclass at loadtime to experiment with adaptable programs. This was circa late 90s. Loads of fun (npi).

What scares many away from Java is that it is now a very huge mental object. But not wanting to admit this, they simply pass along FUD.

> I used to do the switch the superclass at loadtime to experiment with adaptable programs.

This probably falls in undefined behavior territory. What kind of errors did you get when something went wrong?

It is not undefined behavior. Bytecodes are modified via custom CL and superclass was swapped from e.g. Object to something else. As for errors, honestly its been 23+ years so don’t really remember. Naturally you can’t just swap any random class - it has to be correct e.g. can’t have an override method in the child that has no corresponding method in the super. And if memory serves, you can also play this trick using classpaths. Assume a stub package com.foo.Context which is simply an extension of Object, and derive components from com.foo.Context.SuperClass. You simply need to provide a different eponymous package for the adapted instance of the code.

https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.ht...

I've definitely experienced some scary bugs caused by runtime bytecode injection (from a tracing library). I didn't bother tracking it down further than identifying it was at the intersection of some slightly strange type and the tracing. Caused VM errors when this method was called but only when the tracing was configured.

Not saying it's not interesting to experiment with things like that, but there are non-trivial associated, if you ask me, to depending on it in production.

I'm pretty sure (but am hedging here :) that you would get any errors in the verification phase of loading a class. What you saw may have been due to bugs in that library you were using.

Anyway, this was before ASM and other bytecode engineering libraries. Today I would not directly hack the class def myself.

I thought it was understood that choosing a less popular language meant hiring people for fundamentals and then letting them ramp on on your language. Did the client choose Scala thinking they would just hire "Scala people"?

If you stick to a reasonable middle-ground of scala it really shouldn't be hard to teach. I do like the FP community but that style has gone a long way towards mythologizing scala as an impossible language to teach.

In my experience it's clearly considerably simpler than Rust or C++ to become productive in.

I think mixing the OO and FP is the crux of the problem. Type inference and subtyping (class based inheritance) don't mix well, and often require type annotations to help the compiler when types get somewhat complex. Eventually you develop an instinct for it, and it's not a problem, but the road to developing that instinct is littered with torn out hair. (edit: spelling)

I have found that languages that support both FP and OO paradigms its best to do things like data manipulation in FP, and use OO to encapsulate those processes and be limited to just passing messages to other objects.

Avoid inheritance. Once an object is instantiated, don't change its internal state. etc.

>Avoid inheritance. Once an object is instantiated, don't change its internal state. etc.

Just embrace Clojure then. You get all that enforced for you, plus the entire Java eco-system.

I'm not a huge fan of the Java ecosystem.

Clojure does look really nice and Diatomic looks pretty slick.

Have you used Clojerl? https://github.com/clojerl/clojerl

When you take away Java, the JVM ecosystem is actually really nice. It's ok if you don't like it, I'm just saying, I used to think the same, but since taking on Clojure I've completely changed my mind, and find the Java ecosystem one of the best out there.

Clojure's ecosystem is even better, it improves on the Java one a lot, simplifying most of the warts with the Java one, and like you've brought up, it goes beyond the JVM in having quite a lot of active dialects.

I haven't tried Clojerl beyond just REPL, but it seems quite complete, and the maintainer is very active. ClojureScript is also quite nice, if you prefer the JavaScript/Node ecosystem.

You can also use ClojureCLR if you like the .Net ecosystem better.

And finally there's Babashka which deserves a mention, if you want something more like Python.

You also get to play with Clojure derivatives as well: Janet, Fennel, and Ferret can all be picked up in a day if you already know Clojure.

For me Clojure was a great investment, and it's replaced all my needs in all areas: scripting, front-end, application development (mobile and desktop), command line tools, server side, big data, batch processing, ML, data visualization, etc. Only downside is you will exist within a niche, but that niche has everything I need.

nice! i'll check it out!

You lose static typing though, and that's a deal breaker with me.

And no, core and other half-hearted attempts at gradual typing from Clojure don't even come close to hitting that mark.

Yes this is the conclusion I've drawn. Often called "FP in the small, OOP in the large".

Higher-level organizational patterns in pure FP style aren't familiar enough to replace the ergonomics of just using a few classes and DI to make things modular. Maybe tagless final or something similar will eclipse it eventually, but for now it seems like using "anemic style" OOP, with functional in the small is still the best we have for mainstream development.

I agree you should avoid inheritance, but it’s the fact that Scala supports inheritance that makes type inference difficult (also null). Type inference in Haskell is much better, for example. I saw a great presentation on the topic by Thomas Wies. Here’s a paper he wrote if you are interested https://cs.nyu.edu/wies/publ/finding_minimum_type_error_sour...

This is my philosophy:

Data types are object-oriented. They are responsible for ensuring that their internal state is consistent, and nothing more. They may inherit if it makes sense, but it usually doesn't and most of the time aggregation is more appropriate.

Business rules are functional. There's just a bag of composable, pure functions that take the various data types and perform validations, transformations, etc.

External services, such as a database, are contractual. This would be an interface in Java or C# defining the required operations of the service. This allows manipulation of these external services during testing. Unit testing would mock them; integration testing would not.

Workflows or services are procedural. They combine everything else into an actual use-case. They read in a linear flow of what needs to happen when to meet the use-case.

Great breakdown. Seriously, I've had very similar ideas but never tried to build a coherent mapping like this. Thanks for writing it out!

Maybe the best approach is using a language with dynamic types ? (e.g: cloujure)

We chose Scala, had so many problems with the eco system and compiler and community, still love the language, it's the programming language I've enjoyed most, but glad we sold the startup.

Scala has its place... it's just not one that requires fast on-boarding (like a small startup where people constantly come and go). The success stories usually involve companies that are mature enough to spend time on design, training, and supervision (wix, for example)

>We really loved the integration of OO and FP, really miss the FP awesomeness honestly... but simplicity trumps doing either of those things well.

Agreed. I like the Scala language, but I can't imagine a scenario where I would recommend it for a project over Java.

> but simplicity trumps doing either of those things well.

That's the genius (?) of Go. They gave us what we didn't know we needed.

Software people love the "manliness" that comes with "serious", "real" programming languages. Maybe it's the desire to impress our peers with our intelligence? But in the end the only thing that matters is whether you can get your shit done and on time in at least a semi-working state.

I appreciate what you're getting at, but is Go really the epitome of a "get things done" language? It seems to suffer from some serious boilerplate issues even compared to something like Java.

As far as it being driven my machismo, I think that's part of it, but I'd be more inclined to blame it on the following:

* complex languages/mental frameworks function as an "intellectual trap" where people more inclined to be interested in intellectual things for their own sake start to lose site of the forest for the trees.

* FOMO that one must keep up to speed with all the coolest new developments in programming or face mounting irrelevance

* Elitism/self promotion where specializing in something difficult (and potentially keeping newcomers out) either provides internal or external motivation

I think some of these traits seem somewhat correlated with the "nerd machismo" archetype but I think the actual causes are multifaceted. .

Pretty much a noob at FP in general: Wouldn't f# solve the FP and OO integration?

I did Scala for ten years and've been at it with F# for a year and a half now. I do miss some very powerful features of Scala such as HKT and "typeclasses" occasionally but overall it works very well.

Sure, Scala is complex but I find that there are a few misunderstandings here.

Java, for instance, _requires_ that you use a number of supporting tools for it to be useful these days. You need a container, you need quite extensive testing and you need quite the advanced build tools. Sure, these have all been around long enough these days to not be what we think about when we compare Java to other things, but they're implicitly also Java. I can't speak for Go because I've not looked at it.

But Scala has strengths that means that you don't need to have _external_ tools and libraries for those things. That strength is types which means that the compilers is going to want to "pick a fight" with you to a very much higher frequency. That is a pain. The reason that we're here talking about that pain with Scala is that you don't think about that in your Java project anymore because you use heuristics and sandy heads instead.

Another thing is that I think that Scala gives you a big fat revolver to shoot yourself in the foot with. Java has a bb-gun so no matter where you aim it, you're not going to accomplish a lot.

Over to F#: I'm going to claim that the .NET toolchain is superior to the JVM ones. The CLI tools and paket runs circles around the POS that is SBT.

Huh, I use Java all the time without containers. You definitely don't need them.

> integration of OO and FP

OO and FP are paradigms. You can use any of them, or both of them, in any language.

You can use FP in Java, or OO in Haskell. That doesn't mean that they're equally usable paradigms.

I believe Scala had the goal of offering a high level of usability for both FP and OO. It failed despite an admirable effort, because of an incredible amount of language complexity.

I wouldn't blame the language, I'd blame the people who chose Scala.

The impression I always had was that Scala was a research project into how various programming language features could live together and interact, hence their everything and the kitchen sink approach. From an academic point of view I think Scala was a huge success.

That people chose to use Scala in production speaks more to how badly people hate Java and want an alternative, not to Scala's failings.

Indeed, Scala looks like a language research project, more like Perl and Haskell. You definitely can use all of them in production and benefit from that, but you need both mastery and restraint in using such a language.

Golang is the opposite: it takes the ideas from 1980s with some early 1990s (Pascal, Modula-2, Oberon; CSP; garbage collection) and executes them to near perfection. It even ignores some of the obvious faults in ergonomics (if (err != null) return err; everywhere) to keep things dead simple and reasonably performant. It's much harder to write incomprehensible code in Golang, even if you try, so lack of mastery and restraint would have much less catastrophic consequences than in a highly expressive language like Scala (or e.g. C++).

My issue with Golang is that I find my ability to write code to just takes so much longer (and often with more lines) than other contemporary languages like Swift. The rigidity is a double-edged sword.. it removes potential issues that could cause you to trip on yourself, but at the cost of productivity.

Maintaining Haskell code in my experience is s piece of cake. The community is really careful to avoid breaking changes, and when they can't be avoided, to introduce them in the most polite, gradual, convenient way possible.

Recent Javas made many QoL improvements to remove some of the complaints IMO. And there are more neat improvements coming (already there?) with 15 and 17.

Also, Kotlin is a rather nice and much less radical "better Java", well-supported, but mostly popular on mobile.

Kotlin is by far my favorite language-to-date. I moved my dev team to it years ago, it's all server side. We still have over a decade's worth of Java code to maintain, but anything new is in Kotlin (even in the legacy applications, we just add Kotlin). So nice. Makes me happy when other people get onboard and say "oooh yes, this is much better than our legacy Java". I was a HUGE Java proponent from 1999 to around 2010, when I started really getting annoyed that the language was missing so much that other languages had. Eventually we tried Scala, too complex to deal with in large teams to get everyone onboard, Clojure was a pain as everyone tried to be as clever as possible (and errors were awful, tooling was painful as not everyone wants to use emacs), then Kotlin came around and we never looked back.

Kotlin on the server is growing though and has the biggest library ecosystem to exist

Glad to hear this. There's really no reason it for it not to do well on the server.

Pattern matching is (will be) better implemented in Java. It's fully fledged compared to Kotlin's implementation.

When will the last-argument-as-block be implemented in Java?

This one brilliant feature allows for easy DSLs without macros, parsing, or otherwise introducing new concepts to the language.

The ironic thing is, Java now has records (case classes in Scala) and pattern matching (fully fledged Scala-like pattern matching is in the works). It's better designed than Scala, and will be faster due to better integration with the JVM.

> better designed than Scala

How so? Do you mean Java in general, or just the pattern matching part? Can you provide an example where Java is "better designed"?

Destructuring in Scala is allocation heavy, I expect it to be much lighter weight in Java.

Scala compiles to java bytecode. Scala can only be faster then Java, in that Scala can leverage all functionality of the JVM and on top provide extra smarts that Java cannot due to backwards compatibilty it has to offer.

This has never been true for non-native languages (Scala, Clojure, Kotlin, etc.) as far as I'm aware. They've always been slower. The JVM is designed with Java in mind and co-evolves with it.

One of the benefits of Scala's willingness to break bytecode compatibility all the time is that whenever Java/JVM implements a Scala concept, Scala switches to the Java method under the hood. Whenever the JVM gets Java-specific optimisations, Scala will use those. That's honestly pretty cool.

They have in practice been slower because they try to be higher level, more ergonomic languages than Java. There aren't many performance-first JVM languages yet I think (Fortress maybe? They cited JVM problems when they wrapped up though).