As much as I love Clojure I see the problem is you need really good (not just intelligent but also wise) developers to reap benefits. Otherwise you can run into huge problems.
"With great power..."
Clojure has so bad rep here that even I think I made a mistake that I let on that I use it for my projects.
I think your observation that you need really good developers is the key insight to problems that come about from using Clojure. Stu Halloway even mentions it in the interview:
"And what we would quite often see is that people would write a Java app in Clojure, or people would write a Ruby on Rails app in Clojure. And not surprisingly, it would have weaknesses that you associate with idiomatic Java apps or idiomatic Ruby on Rails apps"
Clojure, as a language, makes it very easy to do stuff and sometimes that stuff means making code look like code that only makes sense in other languages. This is a terrible mistake but one that is easy to understand. Very few developers have any real background in functional programming and it is too easy to start hacking things together.
As a longtime Clojure user (dating back to attending the very first Clojure conference in Durham, North Carolina), I think there has been a little bit of "ego-driven" developer syndrome with projects in the language. There is definitely a tendency for new Clojure programmers to get too fancy.
This is not a problem with Clojure the language though. I made the mistake myself in early projects. The more I have used Clojure, the more I have realized that when I am doing it right, I am spending significantly more time understanding the problem I am solving than throwing code together . . .
With Clojure the mess is still 100k to millions LOC because the guys did not know how to make worthwhile abstractions, but now it can't be parsed by IDE and you are screwed trying to figure out what happens at runtime.
ALso, if you think if there is less code then it is easier to understand, go and read any advanced Common Lisp book (like Let over Lambda) and try to really understand what some of the more advanced macros do, exactly. It is definitely not the same as reading pages upon pages of redundant code.
Worse, the abstractions made it so that even a simple feature would require about six files. Not counting the tests.
Not to say that clojure has been a breeze. Worst I see there is developers refusing to use libraries it external programs and instead taking a ridiculous "first principals" approach. Even if they get it working, it is typically not search friendly, as everyone else is using some other tool. Basically, the same trap I see in build systems,
Spring annotations, maybe?
I have an enterprise client right now sitting on 1.5M java 1M SQL 1.5M XML for a single government system that mostly does not function at all, it just gets passed from contractor to contractor, each one sucking down a hundred million dollars before tossing the hot potato. Thanks for the money, Java!
Can't hire for it, can't refactor the code at scale, etc.
Be careful with what you read, newer languages are usually trying to sell themselves where established languages have happy people not trying to sell it, or, resume driven developers trying to sell it as bad.
This is not my experience whatsoever.
I am writing a little bit of C# atm and astonished how you can just dot tab through your work (without ever having read a tutorial or book on C#) and all those other contextual information you get with "full" visual studio. Somehow feels like just letting you guide through the work :).
On the other hand, the students I teach really struggle when they don't have all their IDE tricks. They learn almost exclusively C# at their institution in the first year(s) while my experience is mostly playing with Unity ;).
Frankly, VSCode is reasonably good at giving you something for Python, and the relative lack of static analysis is offset by other qualities of the language, so this feels like a pretty weak argument.
What if Clojure projects are more risky by selection bias?
Now, imagine giving the people environment that imposes no structure on your project and gives hyper powerful tools like macros and you are in a big problem.
At least with Java you get Spring and this is how you do endpoint, this is how you connect to the database, and so on. The structure is suboptimal and redundant but at least they are getting some guidance which are most likely already familiar with.
In Clojure, as a lead of the project, you would have to do the same but now it would be up to you to define all of that guidance. The question is, are you better at this than entire organization that supports Spring?
The only situation I see where Clojure would realistically be beneficial is when you only get like world class developers who know what they are doing and can work together.
I can't work on a Clojure project with a person that is unable to go through something like Let over Lambda with full understanding of the code examples and trust they will be able to make good decisions when building abstractions with macros.
Isn't this a cultural, organizational problem? You said your original developers left or moved on to management and your new hires are rather inexperienced. Wouldn't it be a wiser long term strategy to facilitate learning and knowledge retention through mentoring, documentation, teaching and attractive technical career tracks?
For example there are several popular open source libs/solutions to manage application state and structure declaratively. And it is well known in the Clojure community that macros are a double-edged sword and should be used to solve specific problems.
I agree that Clojure is kind of a second (plus) language. But I also think there is a reasonable path from writing simple Clojure to wielding the more powerful features.
And why, as a developer, you want to be working for a software company where your work is generating revenue, not somewhere where you are a cost to be cut.
Found your problem.
The first time you see a technology you immediately see things that solve some of problems you have, especially because authors tend to market pros more aggressively than cons.
Finding issues with technology takes some experience handling it.
For a relatively new thing that is in growth stage there is disproportionate amount of people who just joined and so have skewed perception.
I also thing people who would write about a product would usually write quite quickly after adoption but then won't follow up later after they got some experience.
There's some doublespeak going on here.
When "leadership" says "maintainable" what they mean is that the project is recoverable after they frog march the project lead out the door, or lay off the team, or the team quits because the job blows. Their view of maintainability is dominated by Bus Factor and similar existential concerns, rather than agility experienced by the project's authors. They actually want the opposite of maintainable: Rigid and heavily specified.
Even there, the projects suffered so much to hit release dates and stay understandable as the system grew that velocity suffered relative to peers doing experiments in traditional OO languages. In fact, the maintenance burden (for code composed by smart senior people!) is so high that it's actually convinced me on the virtue of regular OO/procedural languages over lisp.
This is an example of the problem of Clojure being promoted as Lisp. There are now people whose idea of what is "Lisp" is represented by Clojure.
You might like coding in a normal, procedural Lisp with OOP, but due to a bias induced by Clojure, you don't suspect that's even a thing.
This isn't one and don, people move on even from the best jobs. You know need to keep replacing them, explain to HR why this "req" (requisition - aka job opening) needs them to look at resumes differently. Why they can't pull from their Java pool of resumes, explain to your director why you aren't like all the other teams. It's a lot of friction.
Ok, but how much of that is just them not knowing Clojure? I've found people that get handed down a code base in a different language tend to hack their way around it, instead of like, pausing for a minute, take some time to learn the language, and then come back to it.
Maybe in your case it was a terrible code base of low quality, but I'm just curious.
The worst codebases (and that is true of other languages just as for Clojure) that I have seen are ones done by people who are intelligent enough to learn every feature of their language and any little trick available on the internet but haven't yet had the chance to acquire the wisdom on when to use them or what to use them for.
For those that don’t know, Rich Hickey (and by extension, Clojure) is really big on data-first dynamic typing. So idiomatically you’re supposed to do most of your work in Clojure using their first-rate immutable collections library, primarily maps and their sequence abstraction. Clojure provides some mechanisms for specialized domain objects, but it’s fair to say that this is typically considered non-idiomatic.
This works great if one of the following conditions are true:
1) All data flows through your system on separate tracks. Foo’s come in from foo endpoints and go to the foo database, and very rarely do data sets cross paths. Differences in business logic can be separated by API endpoint, kafka topic, or some other difference that lets you separate the call paths thoroughly.
2) Every type of data in your system looks different, so that you can easily determine whether or not a given piece of data is a foo or a bar once and send it down the right call path in one place.
3) In any case where similar types of data must be treated differently, it’s possible to organize your code in such a way that you only have to build up the cond-tree once, and you can use different call paths to treat the data differently.
All of these were often false for us. We ended up in situations where we had lots of data that looked very similar entering from common endpoints that required different business logic at multiple points in the pipeline. If you’re writing idiomatic Clojure this is the toughest case possible, as you end up littering your code with extremely similar cond-trees, which makes extension and verification unnecessarily hard.
In most other languages, even dynamic ones, you’d solve this via sub-classing to both group common behavior and allow specialization. Clojure provided some tools to make this work, but they were both maintenance nightmares in their own separate ways. My favorite is how the results of defrecord (which makes class-like maps in Clojure) could be treated like a map, except certain map operations could accidentally turn it back from a record to a regular map. That’s quite the foot gun, trust me.
For our team, we decided that pure Java was the way to go. Java by that point had started adding a lot of boilerplate eliminating niceness, including lambdas and streams, and the ability to make honest-to-god subclasses really helped out in our specific domain.
My instinct is that you could solve general "where does this come from" and "where does this go" questions with metadata. I believe that was one of the driving reasons for metadata in the first place.
> 2) Every type of data in your system looks different, so that you can easily determine whether or not a given piece of data is a foo or a bar once and send it down the right call path in one place.
Clojure spec, malli, schema are libraries that deal with this kind of problem. They are very expressive.
> 3) In any case where similar types of data must be treated differently, it’s possible to organize your code in such a way that you only have to build up the cond-tree once, and you can use different call paths to treat the data differently.
A combination of spec (or others) and multimethods (arbitrary dynamic dispatch) would come to mind here as a solution.
I assume these things are known and were discussed. Are there specific trade-offs that didn't work out?
> I assume these things are known and were discussed. Are there specific trade-offs that didn't work out?
As is always the case when a team switches programming language, the problem I described was the straw that broke the camels back for us. But fundamentally the issue in this case was "it sure as heck looks like we're trying to approximate class based dispatch inside of Clojure, this sure as heck is silly. Let's just use a language that actually wants to be used this way".
The general problem you describe was also mentioned in the podcast. Trying to write like language X in language Y. This can also be observed within the Go community; people writing Java or JS style programs in Go, especially early on.
Isn't this where you would use multi-methods? If you can determine what the "type" of data is via `cond` (sounds like home-grown pattern matching?), couldn't you replace these situations with multi-methods that switch on the same conditions?
If you can get away with one multi-method, that's fine. The moment you start needing multiple multi-methods in different points in your pipeline then things begin to suck.
> sounds like home-grown pattern matching
Yup! And if you find yourself constantly writing pattern matches in Clojure, then you probably actually want classes, since that's a much better way to bind both data and behavior together.
That sounds more like an indictment of functional programming rather than Clojure specifically. I’m not a seasoned functional programmer but I often read glowing praise from others about the use of pattern matching in their code.
I’m also not a professional Clojure programmer, but I thought that multi-methods were supposed to be the solution to the expression problem [0] that both class-based inheritance and pattern matching suffer from.
It was my understanding that multi-methods are supposed to allow open extension of behaviour without having to do what you just described; track down every instance, and cover every case.
In the case of multi-methods, couldn’t you just co-locate the method implementations next to the data they are supposed to operate on? You should be able to add/modify behaviours without worrying about what the other data types are doing.
Honest questions here. Just trying to learn more about the real world pros/cons of these approaches.
Also, I agree, it's interesting to hear his experience report.
"In object-oriented languages, it's easy to add new types but difficult to add new operations. Whereas in functional languages, it's easy to add new operations but difficult to add new types".
Put more concretely: if your Java code base is full of one-off classes with little to no sub-classing or multiple implementations of internal interfaces, then Clojure might work for you. But if you have a lot of sub-classes and repeatedly implemented domain specific interfaces I would hesitate mightily before considering Clojure.
My team's domain was such that the operations were very fixed, but the types expanded constantly. There is only a fixed set of things that the back office can possibly do with a bond or a future, but new types of instruments come into existence (or sometimes our awareness) with pretty surprising regularity.
But the article you linked had a slight of hand trick with how they described multi-methods. They only give an example one multi-method. What if you need to emulate a Java interface with M methods on it? Well now you'll need M multi-methods with an implementation for each virtual "class". You quickly see how that falls apart if you need to add a type or an operation, since either way requires you to manually check and make sure that all operations are implemented for all types. Again, it works, but it's error prone.
Defrecord & defprotocol work much better. We ended up trying them and they were ... okay. They work, with some footguns, most of which I can't remember anymore, but if you end up going too far down that road (like we did) you end up asking yourself why you're writing Java in Clojure instead of just writing Java in Java.
I'd say that it's a problem (not indictment!) of Lisp style[0] functional programming, where there's a pretty hard wall between how you organize your data and your functions[1]. If we'd been using Haskell, we could have at least declared a type class for our shared behavior and used that to handle the different parts of behavior in the pipeline.
I say problem with some reservations, because your mileage will vary depending on the domain in question. My current team could use Clojure quite effectively since the domain is much more amenable to how Clojure approaches these problems. My position is less "this doesn't work" and much more "this has some tradebacks you should be aware of".
> I’m not a seasoned functional programmer but I often read glowing praise from others about the use of pattern matching in their code.
Pattern matching is superior to if/else/else if trees, full stop. If you have to have a bunch of if/else trees, you'd rather use pattern matching.
The problem comes when you start repeating the same if/else conditions (not behavior) in multiple places. In a language like Java, you would naturally start wondering if you should make a class or interface to factor this out somehow. In a language like Clojure you don't have nearly as many tools laying around to collect common behavior into shared locations. You can do it, but you'll be happier the less you have to use multi-methods and defrecords.
> I thought that multi-methods were supposed to be the solution to the expression problem
Multi methods are ... ok. There are a few footguns laying around with them, since using them suddenly makes imports side-effectful, but I can and have used them successfully.
The issue is that multi-methods inherently give you one function, and there's no real way to tie several of them together at all. That's great if your domain is organized in such a way that you can use a single multi-method in one spot to break up the dispatch tree. It's less great if you need to use multi-methods multiple times during processing to specialize based on similar traits of the data. You can do it, but it's error prone and you'll end up with a nagging feeling that you're just making a really bad object system using multi-methods.
> It was my understanding that multi-methods are supposed to allow open extension of behaviour without having to do what you just described; track down every instance, and cover every case.
Yes, and it's this open-ended nature that kind of screws you in some cases. If I have four multi-methods that all need to cover the same cases (with different behavior), it's real easy for me to forget one when modifying my code. That same flexibility in adding instances means that there's nothing preventing me from forgetting one too.
> couldn’t you just co-locate the method implementations next to the data they are supposed to operate on
Well, the data came from Bloomberg, so there was nowhere we could put them that would be "next to" the data. We could put them all in one namespace, but that starts getting unwieldy fast. We could separate them out into different namespaces for clarity, but now it's even easier to miss that you've forgotten things. It's a tough trade-off.
Oh, and multi-methods really messed with our dev tools at the time. Hopefully they've fixed it since then, but a lot of the time Cursive Clojure wouldn't refresh them properly, resulting in dozens of REPL restarts for those of us that preferred Cursive. That got old fast.
0 - As long as you pretend CLOS isn't a thing. This isn't a big deal, since lots of people like to pretend that CLOS isn't a thing.
1 - Yes yes, I know "functions are data too", that indeed was a neat trick ... half a century ago. In Lisps the hard wall is in organization; there typically is very little binding the data to functions like you get with objects[0] or even Haskell style typeclasses. Getting the right data into the right function is entirely driven by how functions are called, rather than by the data itself.
It's hard to discuss your particular case, but what I wonder is if there wasn't a different way to model things that would fit in the Clojure data-droven model. And if not, would there be new patterns or even some extension constructs to it that could allow it to fit?
...but just as pyspark is not “pythonic”, clojure spark is super unidomatic to use. ...which is why its languished unused.
I mean, an idiomatic library isn’t impossible; see how databricks has wrapped the pandas api up with its koalas project; you just have invest the time and effort and actually make it.... for what, fundamentally, is a total absence of demand.
or do you mean like first party support for clojure? Thats never going to happen...
Ideally, it would be great if clojure replaced Python and Scala as the defacto language used to interact with Apache Spark. That's probably not likely, though. It's hard enough to get a lot of folks past spark's weirdness without throwing a lisp into the mix. But it does seem like a functional JVM language that's not Scala would be the optimal way to use spark.