They're really smart people, and they had to be extremely capable programmers to get this far, but the embarrassing question is, how would a team of mediocre developers have tackled this problem? They would have picked a mediocre language and probably written a naive, straightforward solution, and after a few months of squashing bugs and developing production workarounds, it would have been a workable, tolerable system. They would have spent the last year working on something else.
The naive solution would have more inherent unreliability, but our vast and intricate solution, designed to scale to the moon and be more self-healing than the T2000, is never going to approach the same reliability as a naive solution, because it's too complex (and the code is too hard for mortals to read and reason about) for us ever to iron out the bugs.
They make a huge deal out of how much safer and easier to reason about our code is thanks to the FP discipline, but why do we have just as many concurrency-related production issues as a typical team using blocking operations and threadpools? Why do we have _more_ incorrectness caused by swallowed errors than I came to expect in projects that relied on exceptions?
I'm still keen on mastering this style of Scala, because I think the benefits can be had, but it annoys me that some of my teammates are happy to use those stated benefits to justify their programming adventures without seeming to care if they actually materialize.
I don't mean to be rude to your colleagues, but almost by implication in what you've said, they're not good programmers. Being a good programmer is nothing to do with leveraging a fancy FP language. It's about delivering working, maintainable, testable code. Whether you write buggy spaghetti code in Go or misuse Scala, its the same root problem.
I've been a Scala dev for five years, worked on some massive projects and it's been a dream. As long as you agree on a style, don't go crazy with the language and use it with discernment, it's a joy to work with. It's a sharp tool though and it takes discernment to know how to wield it appropriately. I don't know if that's a downside, it's more a warning.
That's fair. I guess if by good programmer we mean, fluency with the language and algorithms / abstractions /types etc then perhaps they qualify. I really meant something more like "software engineer" ie. someone who can architect a simple, sane, working solution adhering to the usual best practices of good software construction. Maybe there's a missing piece between being a good coder vs scaling this knowledge up to the application level. The latter is indeed a much rarer skill and too often, people without it influence major decisions.
My pet peeve in this business are supposedly good programmers who get praised despite never having actual results. And it is not like they would be rare.
Because turns out that popping out algorithm exercise code after another into source control doesn't produce a good product. But that's what current interview fad measures, thus it's what it produces.
And my feeling is the same with Clojure and Scala, I'm not a specialist, but code can be written in such a way that I can even contribute to it.
It's always better to write simple, almost dumb code that anyone can understand than to use advanced abstractions from category theory or whatever.
Why? Because every single study or anedocte I've ever heard is like yours: adding complex abstractions to code do NOT make it more reliable. But they do make it much harder to modify, understand, and fix.
FP tought us that unconstrained mutable state is very bad, and that has helped us immensely in the mundane languages of the world (Java, C#, C++, Go). But I think it's time to learn the other lesson from FP: overly complex abstractions are also very bad and should NOT be used willy nilly as they simply have a very low or negative cost-benefit.
I disagree with the general take that "adding complex abstractions to code do NOT make it more reliable". Good use of abstraction makes code easier to write, easier to maintain and easier to extend. Good use of abstraction can make code more concise and more regular while at the same time allowing better diagnostics when you do something wrong.
As a quick example: a generic JSON serialization library that can work with any type is probably a lot nicer to use that one that requires manual reimplementation for every class in your program. It'll be more complicated to write but it's probably well worth it in the end. Similarly a logging system that can abstract over several backends and log levels depending on the environment probably beats having a bunch of if/else every time you want to log a message.
But one needs to remember the old saying: debugging code is harder than writing it, so if you write code that's as smart as you're capable of producing then by definition you're not smart enough to debug it.
This is the reason I used the term "overly complex abstractions", not just "abstractions" in general, of course. Overly complex meaning it's a failure to use "the right abstraction" where you instead go for something much more complex than the proble called for... this is what I understand most people in this thread are referring to when they refer to previous Scala projects they've worked on.
I used to write complex code because I found it elegant. Now I write simple code because I enjoy breaking down a complex problem into its simplest form, which for me means I fully understand the problem.
I still think programmers should work with powerful tools, because "dumb" tools often force you to express things in convoluted ways, and you don't need a powerful language to make a huge mess. (Java proves that you only need single inheritance to produce virtually unlimited amounts of spaghettiness in real-world projects. The difference between Java and Scala is not that Scala enables teams to produce epic piles of FP crap, it's that we long ago stopped being shocked when teams produce epic piles of OO crap.)
I think the programmers doing terrible things with Scala would be doing terrible things in any language, given the chance. What they need is hands-on technical management. When they have a wild idea, they need to be walked through an appropriate engineering decision-making process. They need somebody with the authority to tell them, "Ha ha, yeah, that design with etcd and Kafka and the robot space lasers would friggin' rock, how cool would that be, but on the other hand, we could just stick a REST API in front of a database and you'd be done in two weeks, so let's compare pros and cons."
It does not matter how "simple" each line of code is.
What matters is how "simple" the whole application/solution is.
You can argue that you can write very simple code in Assembler - all instructions are very clearly defined and very simple (well, maybe not anymore in CISC...). Someone can learn them all in a day. The problem is just that now you end up with a lot of code and overall that will be hard to maintain.
Compare it to Javascript: certainly not the language where code is most maintainable or simple, but the end-result is easier to maintain then assembler.
In the end, the choice of programming language is like a the choice of a compression-tool like gzip. The more complex, the more difficult to learn and use (setting library size and various compression settings) but the output will be smaller compared to a simple tool or no compression (Assembler) while containing the same amount of information.
I write lots of F#, and I find that computation expressions (a fancy version of do-notation) makes my code more "dumb" and "simple", despite being an advanced "FP" feature. This is because it pushes the complexity from my business logic and into library code.
With computation expressions:
async {
let! foos = fetchFoos
for foo in foos do
do! launchFoo foo
do! clearFoos
}
Async.bind
fetchFoos
(fun foos ->
Async.bind
(
foos
|> Seq.fold (Async.bind (fun foo -> launchFoo foo)) (Async.just ())
)
(fun () -> clearFoos)
)
I could not manage complexity without this feature.
Sometimes false. If the (more) complex abstractions are in well-tested libraries that haven't been written solely for your code.
> But [complex abstractions] do make [your code] much harder to modify,
Not necessarily. You can separate concerns; you can isolate changes; and you hopefully reduce the amount of code you've written yourself, significantly even.
> [complex abstractions] do make [your code] much harder to understand
This is as likely false as it is true.
> [complex abstractions] do make [your code] much harder to fix
Again, this very much depends. If you're using a well-tested, widely-used external library with those abstractions, it may well be easier to fix your own code.
Lisp, in its persistent failure to achieve world domination, has been teaching us this lesson for over 60 years now!
Infinitely powerful languages are fun and, yes, powerful. But as soon as the project has more than one or two people in it, they also result in runaway complexity that will hurt the product far more than any gains in expressibility.
Because that was the obsession, they ended up spending all their time iterating on this perfect type system and not enough on the product. I think people with that issue are more likely to want to use Scala.
Result types, Either et cetera make it extremely easy to swallow errors by flatMapping thoughtlessly losing the context of where they were - you typically /want/ the call stack when you hit into an exceptional flow.
For example, I find myself hating the Either type, because I feel like there is a socially established convention that one half of the Either is the type that matters, the value that you want, the value that is the point of the computation you're doing, and the other half is a garbage value that should never be directly handled. So I really feel like I should conform to the convention and reserve Either for cases where one of the possible types doesn't matter. But how often is it true that one side of the Either doesn't matter? People want me to encode success/failure in an Either type, but if I do that, are they going to treat failure with the care it deserves?
I often handle Either (and Option) using pattern matching when I feel it's important to give both code paths equal importance and equal visibility in the code, but people change it because flatMap is supposedly more idiomatic, and they believe that eliminating pattern matching from their code is a sign of sophistication.
I feel like this stems from a strong desire among FP folks for the happy path to be the only one visible in the code, and the non-happy path to work by invisible magic. Maybe there are some brilliant programmers who achieve this by careful programming, but there are people mimicking them who seem to rely more on faith than logical analysis. They just flatMap their way through everything and trust that this results in correct behavior for the "less important" cases.
I'm sorry that this turned into a bit of a rant, but I'm entirely fed up with it, and it accounts for a lot of what I dislike about the code I work with on a daily basis.
For unfamiliar topics or when presented with uncommon insight, I believe rants, monologues, even diatribes are actually some of the best things to read.
There's always a tradeoff between making the happy path clear and making the error handling explicit. The whole point of Either is to be a middle ground between "both cases are equal weight and you handle them by pattern matching" (custom ADTs) and "only the happy path is visible, the error path is completely invisible magic" (exceptions). Given that people in Python or Java tend to use exceptions a lot more than they use datatypes, I'd argue that a typical Scala codebase puts more emphasis on actually handling errors than a typical codebase in other languages.
Where each case really is of equal weight, consider using a custom datatype (it's only a couple of lines: sealed trait A, case class B(...) extends A, case class C(...) extends A) rather than Either.
instance Functor (Either a) where -- a is fixed here!
fmap :: (b -> c) -> Either a b -> Either a c
fmap _ (Left l) = Left l
fmap f (Right r) = Right (f r)
You _can_ trivially construct a type where the two labels are swapped; it's just labels, Left and Right aren't intrinsically important, except insofar as they reflect the positions of the type arguments in written text.
data Either' a b = Left b | Right aIsn't this the same as letting exceptions bubble up in a non-FP language?
It is /infuriating/ to be knee-deep trying to work out which of your 30 Eithers failed with a non-descript error.
The Monad instance for Either means that chaining them together with flatMap has a short-circuiting effect and the first failure will stop the rest of the chain from being evaluated. I find this actually makes it easier to know where your errors are happening, and also allows you to centralise your error handling logic.
If you go onto the scaladoc for Either today, you see a stringly-typed Either where they discard the Exception.
Compared to Haskell, it is probably better in some way because you have the proper stacktrace; but it “feels” impure a bit.. In a way Java’s exceptions are already an Either type with the result type and the thrown Exception (with “auto-decomposition”, unless checked exceptions) —- is the advantages like manual management of when mapping/flatmapping happens worth it in your opinion? Nonetheless thanks for the heads up, I might try out Scala again with the exception handling model you mentioned!
Don't get me started with the whole forcing of "lets be python like" initiatives as of late.
It's not that bad though, because you get much less of these kind of errors than in a typical Java program (for example).
My argument is that as a programmer you /choose/ the base lemmas which you're comfortable with - with an exception you're saying for a large swathe of your code, it will assume a lemma. When is /does/ break you get to point the finger at it with a stack trace saying here is where the lemma was broken. As an aside there are nicer ways to do this ala contracts, but exceptions aren't a /bad/ way.
The equivalent of try.toOption is try catch everything and discarding. The equivalent of flatMapping without adding surrounding context is equivalent to try catch rethrowing. Both of these are /much/ too common in FP codebases.
readFromDb()
.flatMap(x => doSomethingElse(x))
.flatMap(x => doSomethingElseAgain(x))
.flatMap(x => onceAgain(x))
One strategy I find does slightly mitigate this is to use future.transformWith[S](f: Try[T] => Future[S]). This at least presents to you the opportunity to think about error handling a bit more consciously, but I still find that the chaining operator approach nudges you away from thinking about error handling because it becomes quickly difficult to read.
It's probably to do with how limited the control structures are when dealing in Futures. You don't have while/if-else/etc, you must lift all your control into flatMaps and iterations are only doable via recursive calls with explicit accumulator state (yuck!) and nesting. So whereas a properly thoughtful treatment of error cases in synchronous code may take 20 or 30 lines of legible code, this gets tranformed in the async style to 20 or 30 levels of nested callbacks and recursive calls which becomes unreadable.
People love the unifying abstractions underneath these types, and they love developing instincts about how to write code based on them, but from an application programming point of view, their instincts are often counterproductive. I don't think people take a mathematical enough viewpoint. The abstractions can't tell you what is important and unimportant. They can't tell you how your code should be shaped. They can't tell you which values should be transformed further and which should be short-circuited. They can't tell you which values deserve to be given a name for readability and which values should be anonymous.
"All these values are monads, so I can combine them with a for expression" is a meaningless statement of a trivial mathematical fact, not a clue about how you should write your code.
I think error handling code can be written in a straightforward style, but it isn't as pretty as people would like. I think the trap they fall into is holding onto elegance while they reach for correctness, instead of holding fast to correctness while they reach for elegance.
Anyway, a pattern I've started to use in Haskell with ExceptT is along the lines of
handledIo = runExceptT $ do
failable <- someIo
can_handle <- anotherIo failable `catchError` handlerOfSubsetOfExceptionsOrRethrows
return $ updateState can_handle
try {
doSomethingElse(x)
doSomethingElseAgain(x)
onceAgain(x)
} catch {
...
}
>You don't have while/if-else/etc
Maybe you are looking for ifM/whileM functions from e.g. here: typelevel.org/cats/api/cats/Monad.html.
I agree that some kind of logical call stack is a very useful thing to have, and I'd recommend implementing something along the lines of https://github.com/lancewalton/treelog that provides it.
In my experience, mediocre developers would stick to a simple language like Python and only use simple Python features... and write a completely impenetrable, 100% coupled and wholly unreadable mess that would make you wish for over-engineered Scala. Don't even get me started on what happens with "simple" distributed computing tools like Hive...
I see people complain about languages like Scala leading to over-engineering and I just don't get it: have you not worked with sub-par Python or JavaScript code? Sure, poorly designed complex solutions have problems, but so do under-abstracted codebases! Bad code is bad code, and I don't think it's meaningful to say that "over-engineering" is categorically worse than "under-engineering".
I actually worked with somebody who wrote pathologically over-complicated Haskell. Had some friction between us over that. His Python code? Somehow even worse.
No serious language I've seen at either end of the abstraction-friendly spectrum (not Haskell/Scala nor Go/Java/Python/etc) puts a meaningful floor on code quality. An inexpressive language can have a low ceiling for how good code is, but restrict or simplify the language however you like and people will still happily write utterly unworkable code.
I've found that languages—especially less common languages thanks to the "nobody gets fired for IBM" effect—get used as scapegoats for not talking about cultural issues. If people on the team are writing clearly poor, over-complicated and bug-riddled code, it means there is some sort of technical leadership shortcoming, and it would not be any better if they were using Java instead. (I mean, have you seen Enterprise™ Java™ codebases?) And, at the same time, it's clear that a Scala codebase with tasteful technical direction and leadership—conveyed through team culture, code reviews, shared expectations... etc—can be absolutely great to work in.
At this point, I'm leery of blaming languages for programming problems without a clear mechanism. "The language lets people do bad things" isn't a real mechanism because all languages do bad things. "The language attracts people who like complexity" seems specious too. (Again, let's not blame languages for hiring and cultural problems!) "The languages defaults incentivize poor code" is a better argument, although it can be a bit fuzzy. And arguments like "language A doesn't have capability B that we need" or "language C allows classes of bugs other languages don't which empirically cause problems" are the most compelling of all.
Here's the thing, if you look at a basic Spring crud app, you are also using Monads, Monoids, Categories, Traverse, etc. but you aren't expressing it in the type system. Seriously go look at modern Spring's flux stuff, it's all there minus the type classes.
I've seen teams that tried to over engineer Spring, teams that tried to over engineer Node applications, and yes there are teams that over engineer Functional style Scala.
All the teams I worked with (and managed) at Verizon leaned pretty heavily into FP style Scala without much over engineering and the experience was extremely pleasant. The only production issues in my three years there I remember were performance related, finding out how to get more throughput or lower latency out of FP Scala. I literally can't remember any 'bugs' that made it to production.
I agree that using Monads, Monoids, etc. isn't necessarily indicative of over engineering in itself. If used well they can make the code clearer/simpler.
I don't use spring or plan to use it. Not quite sure why it needs to be looked up. What are you trying to say?
I don't think the super complex language features should be removed. Li's libs do some crazy stuff under the hood, but provide a clean, Python-like public interface. Most devs aren't that good and complex underlying implementations leak and yield complex public interfaces.
Lots of folks would love Scala codebases that only use 10% of the available language features and none of the complex frameworks. But, like you mentioned, it's a hard language to use responsibly.
That coupled with side-effect free functional style (as much possible without overstretching) uses the power of Scala and keeps the code concise and readable. That is how I use it and find it very pleasurable to write in Scala compared to Java.
[0]: https://docs.scala-lang.org/overviews/scala-book/introductio...
Does this have the same problem in large languages such as C++ though? Where everyone thinks there's an optimal subset of the language, but no one agrees on what that optimal subset is?
For me, Scala is a multi-paradigm language with tons of features. The community just needs different types of style guides for different apps. vars are horrifying for the functional crowd, but cool with me for example.
They already do it: it's called Kotlin.
In fact, in one project my former employer was involved in, one main reason they picked Scala was to, on the one hand, weed out the chaff from their existing team of .net developers (in a "shape up or ship out" kind of fashion), and on the other to weed out the 95% of mediocre Java developers.
This reminded me of a great blurb about Niklaus Wirth's approach to languages:
>Wirth’s philosophy of programming languages is that a complex language is not required to solve a complex problem. On the contrary, he believes that l languages containing complex features whose purposes are to solve complex problems actually hinder the problem solving effort. The idea is that it is difficult enough for a programmer to solve a complex problem without having to also cope with the complexity of the language. Having a simple programming language as a tool reduces the total complexity of the problem to be solved. [0]
[0] Pg.2 http://www.cslab.pepperdine.edu/warford/ComputingFundamental...
This, I feel, is underneath a very large proportion of everything in our industry.
This is scala equivalent of HN/pg lisp folklore.
I had the misfortune of working with basically this same team, on scala (of course).
I argued for maintainable simple technology but scala was too hip to pass, for many.
Still remember one meeting trying to decipher a bug and while reviewing the code in question the lead scala fan said "It is not reasonable to expect to understand what code does by looking at it. I'll need to go research this for a few days."
I hope to never see scala (or its ilk) ever again. Give me the most stable language and environment, where every question is a FAQ and every odd behavior is documented in every book and I never need to fall into the rabbit hole of language traps. Then I can just work on building the product, which is the whole point.
Ah I had the same experience.
That this is allowed to happen just shows that putting naive non-technical managers in charge of coders is a disaster.
https://news.ycombinator.com/item?id=22526189
> The only project at Amazon that uses Clojure is one legacy project in maintenance mode.
I was on a team building good old crud apps using monads, monoids, categories, combinators, effects cats, seamless and bunch of other nonsense that i've now purged from my brain.
You could've easily mistaken our team for a programming language research group at a university.
This is literally the number one reason i would never choose scala. There is no upper bound to amount of stupidity one can indulge in. Our code was so convoluted that even intellij had trouble understanding what the hell was going on and would spit out compilation errors when there were none.
I now work with clojure team and i feel a sense of relief and weight taken off of my head from all the useless stuff i had to learn and use.