My instinct is that you could solve general "where does this come from" and "where does this go" questions with metadata. I believe that was one of the driving reasons for metadata in the first place.
> 2) Every type of data in your system looks different, so that you can easily determine whether or not a given piece of data is a foo or a bar once and send it down the right call path in one place.
Clojure spec, malli, schema are libraries that deal with this kind of problem. They are very expressive.
> 3) In any case where similar types of data must be treated differently, it’s possible to organize your code in such a way that you only have to build up the cond-tree once, and you can use different call paths to treat the data differently.
A combination of spec (or others) and multimethods (arbitrary dynamic dispatch) would come to mind here as a solution.
I assume these things are known and were discussed. Are there specific trade-offs that didn't work out?
> I assume these things are known and were discussed. Are there specific trade-offs that didn't work out?
As is always the case when a team switches programming language, the problem I described was the straw that broke the camels back for us. But fundamentally the issue in this case was "it sure as heck looks like we're trying to approximate class based dispatch inside of Clojure, this sure as heck is silly. Let's just use a language that actually wants to be used this way".
The general problem you describe was also mentioned in the podcast. Trying to write like language X in language Y. This can also be observed within the Go community; people writing Java or JS style programs in Go, especially early on.
Isn't this where you would use multi-methods? If you can determine what the "type" of data is via `cond` (sounds like home-grown pattern matching?), couldn't you replace these situations with multi-methods that switch on the same conditions?
If you can get away with one multi-method, that's fine. The moment you start needing multiple multi-methods in different points in your pipeline then things begin to suck.
> sounds like home-grown pattern matching
Yup! And if you find yourself constantly writing pattern matches in Clojure, then you probably actually want classes, since that's a much better way to bind both data and behavior together.
That sounds more like an indictment of functional programming rather than Clojure specifically. I’m not a seasoned functional programmer but I often read glowing praise from others about the use of pattern matching in their code.
I’m also not a professional Clojure programmer, but I thought that multi-methods were supposed to be the solution to the expression problem [0] that both class-based inheritance and pattern matching suffer from.
It was my understanding that multi-methods are supposed to allow open extension of behaviour without having to do what you just described; track down every instance, and cover every case.
In the case of multi-methods, couldn’t you just co-locate the method implementations next to the data they are supposed to operate on? You should be able to add/modify behaviours without worrying about what the other data types are doing.
Honest questions here. Just trying to learn more about the real world pros/cons of these approaches.
Also, I agree, it's interesting to hear his experience report.
"In object-oriented languages, it's easy to add new types but difficult to add new operations. Whereas in functional languages, it's easy to add new operations but difficult to add new types".
Put more concretely: if your Java code base is full of one-off classes with little to no sub-classing or multiple implementations of internal interfaces, then Clojure might work for you. But if you have a lot of sub-classes and repeatedly implemented domain specific interfaces I would hesitate mightily before considering Clojure.
My team's domain was such that the operations were very fixed, but the types expanded constantly. There is only a fixed set of things that the back office can possibly do with a bond or a future, but new types of instruments come into existence (or sometimes our awareness) with pretty surprising regularity.
But the article you linked had a slight of hand trick with how they described multi-methods. They only give an example one multi-method. What if you need to emulate a Java interface with M methods on it? Well now you'll need M multi-methods with an implementation for each virtual "class". You quickly see how that falls apart if you need to add a type or an operation, since either way requires you to manually check and make sure that all operations are implemented for all types. Again, it works, but it's error prone.
Defrecord & defprotocol work much better. We ended up trying them and they were ... okay. They work, with some footguns, most of which I can't remember anymore, but if you end up going too far down that road (like we did) you end up asking yourself why you're writing Java in Clojure instead of just writing Java in Java.
I'd say that it's a problem (not indictment!) of Lisp style[0] functional programming, where there's a pretty hard wall between how you organize your data and your functions[1]. If we'd been using Haskell, we could have at least declared a type class for our shared behavior and used that to handle the different parts of behavior in the pipeline.
I say problem with some reservations, because your mileage will vary depending on the domain in question. My current team could use Clojure quite effectively since the domain is much more amenable to how Clojure approaches these problems. My position is less "this doesn't work" and much more "this has some tradebacks you should be aware of".
> I’m not a seasoned functional programmer but I often read glowing praise from others about the use of pattern matching in their code.
Pattern matching is superior to if/else/else if trees, full stop. If you have to have a bunch of if/else trees, you'd rather use pattern matching.
The problem comes when you start repeating the same if/else conditions (not behavior) in multiple places. In a language like Java, you would naturally start wondering if you should make a class or interface to factor this out somehow. In a language like Clojure you don't have nearly as many tools laying around to collect common behavior into shared locations. You can do it, but you'll be happier the less you have to use multi-methods and defrecords.
> I thought that multi-methods were supposed to be the solution to the expression problem
Multi methods are ... ok. There are a few footguns laying around with them, since using them suddenly makes imports side-effectful, but I can and have used them successfully.
The issue is that multi-methods inherently give you one function, and there's no real way to tie several of them together at all. That's great if your domain is organized in such a way that you can use a single multi-method in one spot to break up the dispatch tree. It's less great if you need to use multi-methods multiple times during processing to specialize based on similar traits of the data. You can do it, but it's error prone and you'll end up with a nagging feeling that you're just making a really bad object system using multi-methods.
> It was my understanding that multi-methods are supposed to allow open extension of behaviour without having to do what you just described; track down every instance, and cover every case.
Yes, and it's this open-ended nature that kind of screws you in some cases. If I have four multi-methods that all need to cover the same cases (with different behavior), it's real easy for me to forget one when modifying my code. That same flexibility in adding instances means that there's nothing preventing me from forgetting one too.
> couldn’t you just co-locate the method implementations next to the data they are supposed to operate on
Well, the data came from Bloomberg, so there was nowhere we could put them that would be "next to" the data. We could put them all in one namespace, but that starts getting unwieldy fast. We could separate them out into different namespaces for clarity, but now it's even easier to miss that you've forgotten things. It's a tough trade-off.
Oh, and multi-methods really messed with our dev tools at the time. Hopefully they've fixed it since then, but a lot of the time Cursive Clojure wouldn't refresh them properly, resulting in dozens of REPL restarts for those of us that preferred Cursive. That got old fast.
0 - As long as you pretend CLOS isn't a thing. This isn't a big deal, since lots of people like to pretend that CLOS isn't a thing.
1 - Yes yes, I know "functions are data too", that indeed was a neat trick ... half a century ago. In Lisps the hard wall is in organization; there typically is very little binding the data to functions like you get with objects[0] or even Haskell style typeclasses. Getting the right data into the right function is entirely driven by how functions are called, rather than by the data itself.
It's hard to discuss your particular case, but what I wonder is if there wasn't a different way to model things that would fit in the Clojure data-droven model. And if not, would there be new patterns or even some extension constructs to it that could allow it to fit?
For those that don’t know, Rich Hickey (and by extension, Clojure) is really big on data-first dynamic typing. So idiomatically you’re supposed to do most of your work in Clojure using their first-rate immutable collections library, primarily maps and their sequence abstraction. Clojure provides some mechanisms for specialized domain objects, but it’s fair to say that this is typically considered non-idiomatic.
This works great if one of the following conditions are true:
1) All data flows through your system on separate tracks. Foo’s come in from foo endpoints and go to the foo database, and very rarely do data sets cross paths. Differences in business logic can be separated by API endpoint, kafka topic, or some other difference that lets you separate the call paths thoroughly.
2) Every type of data in your system looks different, so that you can easily determine whether or not a given piece of data is a foo or a bar once and send it down the right call path in one place.
3) In any case where similar types of data must be treated differently, it’s possible to organize your code in such a way that you only have to build up the cond-tree once, and you can use different call paths to treat the data differently.
All of these were often false for us. We ended up in situations where we had lots of data that looked very similar entering from common endpoints that required different business logic at multiple points in the pipeline. If you’re writing idiomatic Clojure this is the toughest case possible, as you end up littering your code with extremely similar cond-trees, which makes extension and verification unnecessarily hard.
In most other languages, even dynamic ones, you’d solve this via sub-classing to both group common behavior and allow specialization. Clojure provided some tools to make this work, but they were both maintenance nightmares in their own separate ways. My favorite is how the results of defrecord (which makes class-like maps in Clojure) could be treated like a map, except certain map operations could accidentally turn it back from a record to a regular map. That’s quite the foot gun, trust me.
For our team, we decided that pure Java was the way to go. Java by that point had started adding a lot of boilerplate eliminating niceness, including lambdas and streams, and the ability to make honest-to-god subclasses really helped out in our specific domain.