"higher-kinded types, which are immensely useful once you get used to them"

Could you give an example of such immense usefulnes?

All of the problems that aspect-oriented programming tries to solve, but without any reflectioney magic. So for example I have a custom type to represent database operations that need to happen within a transaction; the type system ensures that there will be a transaction boundary around them eventually, but I can still compose together several different such functions and run a single transaction around them all. In theory you could do this with a "command object", but in a language without HKT you'd have to reimplement all the utility functions that make it practical to use those objects (e.g. traverse, which takes a list of must-happen-in-transaction commands and combines them into a single command that will evaluate to a list) and in practice people doing this kind of thing in Java/C#/Python/... give up and fall back to reflection/AOP/metaclasses/decorators, because it's too cumbersome to manage the transaction boundaries explicitly. But then you get things like http://thecodelesscode.com/case/211 happening.

With HKT you can have reusable, generic functions that work on any kind of effect - all the examples on https://philipnilsson.github.io/Badness10k/escaping-hell-wit... and more. E.g. a tree structure library will offer a traversal method that can already handle doing an effectful operation at each level and composing together the effects correctly - even if it's an effect that didn't exist when that library was written. That makes it practical to manage these effects/cross-cutting concerns explicitly, which enables fearless refactoring (it's always clear whether you can e.g. reorder some statements without changing behaviour or not), so you need much less test coverage for the same level of confidence which in turn makes refactoring even easier, and you end up with a virtuous cycle where your code stays clear because it's always easy to refactor for clarity.

Thank you, I've read the links, but without an example of Haskell/Scala code the benefits for any particular real problem are not clear to me.

Specifically, speaking of the second link, what happens if you need to get rid of two types of 'hell' at the same time?

> Thank you, I've read the links, but without an example of Haskell/Scala code the benefits for any particular real problem are not clear to me.

Do you have cases where you've used AOP/reflection/metaclasses/decorators/macros or similar "magic"? (Exceptions or mutable variables sort of work as examples, but most people don't have experience of working on codebases that don't have them). It's hard to give simple examples because these things exist to solve complex problems - one should never reach for a monad where a plain function will do.

> Specifically, speaking of the second link, what happens if you need to get rid of two types of 'hell' at the same time?

There are several possibilities. First, it's worth noticing that most forms of "hell" are fine on their own, it's the interaction between two that tends to be surprising - e.g. https://glyph.twistedmatrix.com/2014/02/unyielding.html talks about how important it is to explicitly manage async, but it's making an underlying assumption that your language will have mutable variables. Unmanaged, pervasive mutation in a language without async is ok; equally, unmanaged, pervasive async is ok in a language without mutation. So often you can get away with managing one effect and letting the other happen naturally according to the program control flow. (This is less true in Haskell because laziness is built into the language, so Haskell has kind of "spent its freebie" there - but it can be a very effective style in Scala).

Second, if you need to work with a particular combination of effects then most effects are available as "transformers" which can be stacked together and the monadic functionality operates the usual way on the whole stack. E.g. several of my applications work with a type that's something like EitherT[WriterT[Task, Seq[AuditEntry], ?], ApplicationError, ?] - an async operation that writes entries to an audit log and might fail with a constrained set of possible failures. Within a single application you usually have the same pattern of effects that you want to manage it, so you can give your "carrier" type a friendly alias and write a handful of helper methods (which are really just simple combinations of standard lifts, but they get used so often that it's worth having short and domain-appropriate names).

Finally if you're writing a library that needs to be reused with different "effect stacks" the usual approach is mtl-style typeclasses (though there are alternatives - I prefer free coproducts). You write functions in terms of a generic F[_] type (that will actually be one of those concrete effect stacks) with typeclasses that represent each "capability" of an effect that you want to be able to handle. This sounds complicated but it's actually the normal way of writing generic functions in Haskell or Rust (which don't have inheritance), and it's very easy at the point where you're writing the business logic: you just write a big "do" block doing all the things you want to do, and e.g. if you need to call "async" then you add "MonadEffect Async" to the function signature.

Of course there's no free lunch - ultimately you will have to define the details of how the async effect gets interleaved with your other effects somewhere, but this technique lets you move that decision out of the library and into the application that's calling the library: there are several different stacks that can be constructed that will conform to the given interface, and the details of how you stack the effects define how the composition will play out. For example if you have a function that both emits log events and could fail, then you can call that with EitherT[WriterT[...], ...] (the log events are always recorded even after the computation fails), with WriterT[EitherT[...], ...] (a failure doesn't include any log events), or with a custom implementation that did something special like marking the last event before a failure in some way.

https://www.parsonsmatt.org/2018/03/22/three_layer_haskell_c... takes a somewhat different approach to application design from what I do, but it's a more comprehensive example of how everything fits together in a full application.

Thank you, I was thinking about your own example with operations and transaction from the previous comment. How would the code look in your approach?

Unfortunately, I have very basic knowledge of Haskell and can't appreciate the last link, however I've got the general idea of the second part of your comment (I think, maybe not) and two things bother me:

1. What if I find it convenient to two use different effect stacks in two different parts of my app? How do I write the code that call functions from both parts of the app?

2. Suppose I use the same 'carrier' type in the app and one day want to handle one more effect, what changes to the code using this type does this entail?

> How would the code look in your approach?

We'd use "do notation" in Haskell, or "for/yield" in Scala. So in Haskell it would look like any of the "escaping hell with monads" examples, or in Scala the code might look like:

    for {
      user <- loadUserForUpdate()
      groups = calculateGroupsToRemove(user)
      _ <- groups foldMapM {group => remove(user, group)}
    } yield groups

We can see that calculateGroupsToRemove does not participate in the transaction but loadUser and remove do. loadUserForUpdate might return MustHappenInTransaction[User] whereas calculateGroupsToRemove just returns a Vector[Group], but it's useful to be able to see the distinction at the call site. Note that we have to use the (standard/well-known) "foldMapM" function to map over the list of groups, instead of "foldMap". You can't write these generic helper functions in a language that doesn't have higher-kinded types, and without them this technique would be too cumbersome to use in practice.

The code evaluates to a MustHappenInTransaction[Vector[Group]] or something like that, and eventually at some point we have an unsafePerformTransaction function that actually runs the transaction. It's still possible to mix up transaction boundaries by calling unsafePerformTransaction multiple times, but, like with the "unyielding" example, it's an opportunity to notice any mistakes.

> 1. What if I find it convenient to two use different effect stacks in two different parts of my app? How do I write the code that call functions from both parts of the app?

To write functions that you can use from both parts of the app, you'd need to use the same approach as if you were writing a library (i.e. MTL-style typeclasess). Then at the use sites you just call the functions - it's actually just generics. Like, if you have a generic sort[T: Sortable] function, you can call that either from another generic [T: Sortable] function, or you can call it with a concrete type like String (as long as that type is sortable). In the same way, if you have an "effect-generic" function you can call it either from another effect-generic function with (a superset of) the same constraints, or you can call it from a concrete effect stack (provided that concrete stack satisfies those constraints).

> 2. Suppose I use the same 'carrier' type in the app and one day want to handle one more effect, what changes to the code using this type does this entail?

If you're using the "single global carrier" approach, what I'd do is redefine the type alias, reimplement the "helper" methods to do one more lift, and then code that's just using the carrier will stay the same. E.g. when I add the audit entry functionality, I change

    type ApplicationStep[A] = EitherT[Task, ApplicationError, ?]
    def wrapSuccess[A](t: Task[A]): ApplicationStep[A] = t.lift[EitherT[?, ApplicationError, ?]]
    ...

to

    type ApplicationStep[A] = EitherT[WriterT[Task, Seq[AuditEntry], ?], ApplicationError, A]
    def wrapSuccess[A](t: Task[A]): ApplicationStep[A] = t.lift[WriterT[?, Seq[AuditEntry], ?].lift[EitherT[?, ApplicationError, ?]]
    ...
    def wrapAuditEntry[A](ae: AuditEntry): ApplicationStep[A] = WriterT.log(Seq(ae)).lift[EitherT[?, ApplicationError, ?]]

and then code that uses the existing type and helpers remains the same, and new code can use the new helper. Code that works directly with EitherT would need similar changes (extra .lift call).

If you're using the "mtl-style typeclasses" approach then you don't need to make any changes to code that doesn't emit the new effect. Functions that want to write audit events will need an additional ": MonadTell[AuditEntry]" constraint, which will ripple up to functions that call those functions, and then at top level the type used at the entry point has to change.

Well written answer, thanks for sharing