This is spot on! Way too often in Scala codebases, which tend to wrap IO calls in Future or similar is that you'll have chains of side effecting futures like

    readFromDb()
      .flatMap(x => doSomethingElse(x))
      .flatMap(x => doSomethingElseAgain(x))
      .flatMap(x => onceAgain(x))

And this all gets passed up the call stack to one generic thing that basically does nothing in the case of error, maybe logging it and that's it. It's tempting to write code like this because it's so easy and looks so clean and readable, but in reality it is not how you build reliable software because what one must do in response to a failure from the first call is not the same as what one must do for a failure of the second call, or third or fourth, but this flatMap().flatMap().flatMap() style seduces the programmer into thinking they've handled errors when really they've just lumped the entire workflow as one big chain that can fail at any point and in practice usually just ignore all error cases.

One strategy I find does slightly mitigate this is to use future.transformWith[S](f: Try[T] => Future[S]). This at least presents to you the opportunity to think about error handling a bit more consciously, but I still find that the chaining operator approach nudges you away from thinking about error handling because it becomes quickly difficult to read.

It's probably to do with how limited the control structures are when dealing in Futures. You don't have while/if-else/etc, you must lift all your control into flatMaps and iterations are only doable via recursive calls with explicit accumulator state (yuck!) and nesting. So whereas a properly thoughtful treatment of error cases in synchronous code may take 20 or 30 lines of legible code, this gets tranformed in the async style to 20 or 30 levels of nested callbacks and recursive calls which becomes unreadable.

Amen. I saw exactly the same problem with over-reliance on flatMap, either explicitly or in for expressions, in future-heavy Akka code that I do in cats and cats-effect code.

People love the unifying abstractions underneath these types, and they love developing instincts about how to write code based on them, but from an application programming point of view, their instincts are often counterproductive. I don't think people take a mathematical enough viewpoint. The abstractions can't tell you what is important and unimportant. They can't tell you how your code should be shaped. They can't tell you which values should be transformed further and which should be short-circuited. They can't tell you which values deserve to be given a name for readability and which values should be anonymous.

"All these values are monads, so I can combine them with a for expression" is a meaningless statement of a trivial mathematical fact, not a clue about how you should write your code.

I think error handling code can be written in a straightforward style, but it isn't as pretty as people would like. I think the trap they fall into is holding onto elegance while they reach for correctness, instead of holding fast to correctness while they reach for elegance.

That's a great observation about for-comprehensions. It's an other one of these cute niceties that in the most basic cases can come in handy but again nudges users away from consciously thinking through what they're doing (good luck handling the error path in a for-comprehension over Futures).

I dunno. You see the same pattern with chained class methods in Python and it suffers from the same exception specificity problems, but doing it is still a fair judgment call. Sometimes all you need to know is that there was an error in the chain. Like if an exception is raised by dataframe.transpose().to_dict().values() I don’t ask myself where in the chain the error occurred, I just think, “oh that’s weird why couldn’t I turn that into a list of dicts? Did I call those methods on the wrong object?”

Your point's well taken. I'd say that there is a bigger issue in the case of flatMapping side-effectful monads because they're typically dealing with things like writing/reading from databases and the error cases should be thoughtfully considered (do I have to roll something back or perform compensating actions, or clean up anything?)

If understood “flatmapping monads” as anything other than the name of a band I’m sure I would agree.

Doesn't Scala have transformer libraries like ExceptT in Haskell. Aside, much of what this comment thread is about, is similar in Haskell-land as well.

Anyway, a pattern I've started to use in Haskell with ExceptT is along the lines of

    handledIo = runExceptT $ do
      failable <- someIo
      can_handle <- anotherIo failable `catchError` handlerOfSubsetOfExceptionsOrRethrows
      return $ updateState can_handle

So the code still remains legible (minus the plethora of Haskell operators, data wrappers and unwrappers :)), I can trap individual exceptions along the way, which I can handle. And at the end unhandled exceptions are returned by the function as Left.

How is it different from

  try {
    doSomethingElse(x)
    doSomethingElseAgain(x)
    onceAgain(x)
  } catch {
    ...
  }

where each method just returns Unit or throws exception? The version with IO/Future is superior since it at least explicitly states that you can get an error here. If you want to say that Go-style error handling is better because if forces errors handling, I can kind of buy it, but it also has some cost.

>You don't have while/if-else/etc

Maybe you are looking for ifM/whileM functions from e.g. here: typelevel.org/cats/api/cats/Monad.html.