A more contrived example would be a situation in which some piece of data (e.g. a worker's monthly timesheet report) is passed between modules of a programming system, but the receiving module, upon performing validation, discovers e.g. that the employee was working during a holiday.

Handling that situation is hard, since there are multiple ways of handling it, each of them valid in its own specific context. If the employee is in another country where that day is not a holiday, we should proceed without any other actions. If the employee has an agreement with their manager that they are having crunch time, then the system should proceed after applying overtime payment. If the employee is on a flexible time schedule, we should proceed and log this somewhere else; if the employee has no justification for that overtime, we should abort and signal an error; more examples follow.

In other words, when we signal the condition, we do not have full information about what should happen to it. We have the question "What should we do with this timesheet?" and the answer to that question is "It depends."

It is possible to model this situation by a condition type named e.g. EMPLOYEE-WORKING-DURING-HOLIDAY, and instances of that condition type being signaled inside the programming module. We create a dynamic environment where the proper handler routines for EMPLOYEE-WORKING-DURING-HOLIDAY are established, and we call the module's validator function inside that environment.

This process fully decouples the act of signaling a condition from the act of choosing whether to handle that situation and also from the act of choosing how to handle that situation.

One can (and should) document the condition type in the design specifications, and also describe functions that are allowed to signal it.

The described EMPLOYEE-WORKING-DURING-HOLIDAY is akin to a system has a slot to register a callback (or signal which can be connected to receiver, to use a Qt-like design) which can handle the situation or decide raise an exception.

Again, I understand that it might be attractive to have a single unified system to handle the different possible situations instead of separate systems. I find it hard to imagine a case where the decision to be an exception-like, callback-like or restart-like is chosen dynamically at run-time by any or all participants. And like I said, I' d be afraid that if such a case come up, it would make understanding the design harder, not simpler.

For example, as much as Qt signal/slot mechanism is powerful and flexible, I've found that when used fully, it makes understanding the code hard because it becomes impossible to know what will happen when a signal is raised because the handling is so well uncoupled.

The difference is the fact that Common Lisp has facilities that allow two things: choices of what and how to proceed and flexible non-local returns. The callback is allowed to list all recovery choices that are present in a given dynamic environment; they are established dynamically, just like handlers, and therefore can be provided fully from outside.

For instance, a handler/callback can invoke a choice named CONTINUE if it decides that absolutely nothing needs to be done and the validation is safe to proceed. The validating code doesn't need to know why exactly that choice was taken.

Or, if it notices that e.g. it needs to convert a timesheet from version 3.0 to version 4.0, it can call a conversion function on the timesheet object and invoke a choice named REVALIDATE, passing the converted timesheet as an argument. Note that the validating code doesn't need to know about any details of the conversion routine! It only needs to provide a means of restarting the validation by establishing a REVALIDATE choice.

Or, if it decides that the situation is hopeless, it can signal an error of its own and defer the responsibility of handling that situation higher up - all the way to the system debugger, if necessary. The validating code doesn't need to know about any details of why an error was signaled! That's a ton of modularity that we've just given there.

These choices are actualy named restarts in Common Lisp. I purposefully name them choices, though, as I introduce them in the book.

Not necessarily. They may return to any point on the stack that has been "announced" as a valid return point. One can compute a list of these return points, choose which one to utilize, and under which conditions.

That's what constitutes the power of restarts (and non-local exits) in Common Lisp.

I think that in a multithreaded environment you can try to hang a particular thread by forcing it into the debugger in order to preserve all stack information inside it, but I don't know if that is feasible in the long run on production systems since it opens a gate to possibly DoSing the machine.

I do this with threads but my system is just a chat bot, so it doesn't really ever turn into a problem.

But yea, conditions weren't designed from the start with distributed systems in mind, so you almost need a runtime that can save a whole thread to disk and recover it later. Back in the day though you could just open a window and ask the user what to do directly.

I think it depends on your OS. IIRC, on Windows, it was fairly easy to destabilize the OS by spawning threads in a loop and starving other processes on the system. I haven't used Windows in a long while though, so I cannot say for sure.

Firstly, for pretty much what the parent describes. Essentially, to help you continue debugging in the presence of incompatible caller/callees, typos etc. Even with very disciplined developers, these are pretty common in image based, dynamic environments (Smalltalk has similar issues in my experience). Re-starting from scratch would be exceptionally painful so a "patch and continue" approach is very useful. I fully concur that this isn't a compelling use case even in weakly typed languages.

Secondly, for exception throwing and catching of similar ilk to Java/C# etc. In other words, in cases where a less general condition system would have worked as well.

Thirdly, where there are 2 or 3 reasonable options to recover from an error condition, generated in library code, and these are coded with the error. The policy regarding how to handle the error is then made at the application level. You could do something similar without the CL condition system, say by passing flags around (or using globals) and switching on them, but this is super cumbersome/ugly and I don't think I've ever seen it used much.

But honestly, there were maybe 2 or 3 cases where the third use wasn't better coded as "try this one thing and, if that doesn't work either, bail".

So my assessment is that it's not crossed over to other languages mostly because of its limited usefulness in those environments. I contrast this with garbage collection and compile time coding which have more general utility and have successfully crossed over.

Multiple dispatch hasn't crossed over much either (the obvious exception being Julia via Dylan). Now, I did happen to find that very useful so I'm a little more surprised at that than I am at the generalised condition system.

I'm trying to read in some Lisp code, from a wide variety of sources, as grist for a mutational fuzz tester. I don't want to have to define all the packages used in this code. So, when I come across FOO::BAR I want to be able to gracefully handle this rather than causing a reader error if the FOO package is undefined.

Restarts let me do this. In SBCL, there's a restart that let's the handler use a different package in this case. In my case, I just stuck the symbol in some other default package.

One could also imagine this being used as a load-on-demand system. The handler could look up FOO in some list of packages provided by various libraries and automatically load the right library, then return control to the reader to try now that the package was defined.

The reader itself doesn't have to know about any of these various ways of dealing with reader errors; it just has to provide the restarts so the various handlers can tell it how to proceed.

First, I describe how the condition system can be used to work with conditions that are not errors and therefore do not require to be handled; Peter Seibel calls SIGNAL a "primitive" function, therefore completely skipping the scenario in which a condition may not be handled. And, by that, he skips non-error non-warning conditions altogether!

Second, the PCL chapter does not describe the internals of a condition system, how the handlers and restarts are constructed. My book describes that in detail twice: once, building a sample condition-like system piecewise, and then again, describing the implementation of a complete, ANSI-compliant condition system.

http://jacek.zlydach.pl/blog/2019-07-24-algebraic-effects-yo...

The first part uses an example that you'd probably find not all that convincing (though it's a reworded example from Dan Abramov's post). The second one is not about error handling at all - but instead, shows how you can use non-error conditions to bolt on a pseudo-UI on top of an operation that signals appropriately, using both restarts (to abort the operation) and signal handlers (as sort of observer pattern).

I'd say the magic of CL's condition system is in restarts, and the ability to choose them programmatically. It's a powerful tool in API design, that allows you to cheaply expose out-of-band interface for monitoring and error recovery (and I mean actual recovery).

Suppose you're writing a module that extract data from a bunch of files (possibly accessed over network). In the high-level view, you have two layers there: one that loops over files, and the other that goes over data in a given file and extracts values of interest.

Writing this in Common Lisp, you could define a restart in the loop layer, allowing to retry a download, or substitute a different file URL. Then, in the file processing layer, you could define a restart around reading an invalid value, allowing to skip it or substitute it. These error conditions and restarts are now part of your module API. The users of that module could then choose different strategies for error recovery. In one case, if file access fails, you need to abort everything. In another, you need to skip it. In yet another, the caller knows of a backup data file, and can pass that information to the restart. Similarly, invalid values could be replaced by an appropriate default value, again provided by the caller in the context.

Picked apart I just can't see the value. You're hitting an unexpected condition then trying to handle the resultant exception intelligently, whereas you should have considered that possibility up-front and handled it non-exceptionally.

The only interesting part is for the error handler to be delegated to a human, but in a way that suggests to me that you've not done your analysis/testing/speccing properly. In a normal system I'd have something like a dead-letter endpoint for such an unrecoverable delegate-to-the-human condition. I can see the extra flexibility that would bring I admit but at what cost of gaping holes?

I feel more and more I'm missing something here.

That is the inverse of the idea. The only possibility we consider up front is that we might need to restart the process of validating the timesheet from the scratch. Why we might need to restart it, and with what data we should restart the validation, we don't really know - we let the caller specify that. If they decide that they want to make use of that restart strategy, we have successfully delegated the responsibility of making that choice and providing data for that restart elsewhere - to the caller.

We are therefore capable of decoupling the handlers and restarts from the real validation code. This means that the caller of the validation function can specify their recovery strategies at the calling site of the function or higher in the stack above it, and also that they can decide to choose to make only some of these strategies available in a context where other strategies simply do not apply.

This makes no sense. Unless you either a) alter the code or b) delegate to a human somehow[0], restarting the process is pointless - you'll get the same outcome.

> If they decide that they want to make use of that restart strategy, we have successfully delegated the responsibility of making that choice and providing data for that restart elsewhere - to the caller.

So get the caller to give that same info first time, and the exception won't happen because the exceptional condition has already been accounted for, by "provid[ing] data" (that necessary data) the first time round.

[0] which you mentioned and I did find interesting

To quote a famous person, “insanity is doing the same thing, over and over again, but expecting different results.” Hence, when we restart in this scenario, we are not restarting blindly in hope something changes; we use the restart logic to alter the input to the validator function.

* We call the validator, which in S-expressions looks like (VALIDATE-TIMESHEET TIMESHEET). This signals an error and invokes condition handlers matching that error.

* Some condition handler notices that this is a timesheet in an old format, computes an updated version of the timesheet that we will call TIMESHEET2, and invokes a restart named REVALIDATE with TIMESHEET2 passed as an argument.

* This restart sets the TIMESHEET variable to the modified value of TIMESHEET2 and transfers control to a point just before the validator is invoked.

* And so, (VALIDATE-TIMESHEET TIMESHEET) is called again, but this time TIMESHEET is in a new format, so no error is signaled and validation progresses.

All of the above can obviously be done statically, but the additional value is in decoupling the concrete means of handling the condition from the point of signaling it. The code that calls VALIDATE-TIMESHEET has no idea

------

If you consider that to be a good idea, I can write an article that gives a concrete code example of this behavior - that should be even more clear than the description that we have here. Should I?

> Some condition handler notices that this is a timesheet in an old format, computes an updated version of the timesheet that we will call TIMESHEET2, and invokes a restart named REVALIDATE with TIMESHEET2 passed as an argument.

So why not just say

  if newformat(ts) do processItTheNewWay(ts)
  else if oldformat(ts) do 
    newts = compute_updated-version-of-timesheet(ts)
    processnewformatTS(newts)

you have the code for compute_updated-version-of-timesheet because you said "Some condition handler [..] computes an updated version of the timesheet that we will call TIMESHEET2" so the code must already exist.

So why expect exceptions when you can anticipate some inputs are reasonably going to cause exceptions and simply handle them upfront?

The only new thing I can think of now is that you can dynamically load a newly-written exception handler (compute_updated-version-of-timesheet, which you've rush-written because you only just found out about the old style timesheets) into a already running program to handle this unexpected condition.

The approach using restarts and the IF-based approach you've listed are fully equivalent in their functioning. I'll claim, though, that if one is capable of computing these situations up front and preparing the recovery strategies ahead of time, then we're no longer doing exception handling whatsoever. If you are able to compute all situations up front, then you don't have exceptional situations, so you don't have exceptions and therefore don't need need exception handling! If anything, one needs proper flow control; these two terms have become wrongly synonymous in some contexts. That's also how I understand your quote of "why expect exceptions".

The timesheet example is very easy to understand because we know everything about the situation: there's a new version that can be an input, an old version that also can be an input, and only one of them does not signal an error. This is why one may also decide to solve this example by using a strong type system - by defining different types (e.g. in Haskell) or classes (e.g. in Lisp) for distinct versions of the timesheet. If you can successfully use the "parse, don't validate" pattern commonly found in strictly typed strictly functional code, then one doesn't need as many exceptions because many exceptional situations become simply unrepresentable in code.

Your example assumes that one can easily couple the means of signaling a condition (conditionalizing on newFormat(ts)) and the means of handling it (calling compute-updated-version-of-timesheet(ts)) inside a singular block of code. That doesn't need to always be the case. Let's suppose that we've just run into the error and need to fix it - the situation that you described, an interactive hotfix of a running system. Fixing this properly requires modification of that piece of code that signaled an error, which requires digging into that module which we might not even have sources for, or we have the source but it's completely foreign to us and we can't afford debugging it on the spot.

We likely might need to develop a patch for that module where we now know that it has a fault and we may also need to live with it for X days before it is fixed upstream. When we have error-handling information stored in the dynamic environment and restart points sprayed across our stack, we have a situation where the error can be handled both without recompiling that faulting module and without destroying the stack - computation can continue when we've provided a proper recovery routine. We aren't getting crashes, stacks are not getting completely unwound.

So you can think of the condition system, or its combo of handlers and restarts, as an intricate system of callbacks and restart points. Every place that calls SIGNAL is a point that can execute arbitary hooks/callbacks/code; every place on the stack that accepts non-local returns is a possible place that can computation can return to in order to resume the computation. This, along with the fact that Lisp is an interactive system (a lot of Lisp's power simply disappears when it's used like Java or Rust or any other batch language), allows for a lot of possible choices of how to handle This New Situation™ that just landed us in the debugger. We can fix it interactively, and possibly store this fix in a new condition handler that we then patch on top of our running program.

Of course, this hotfix might then end up getting factored into the code properly, and what was once a toplevel condition handler that processes some data invokes some restarts becomes a proper part of the code: that situation is no longer unexpected or exceptional we are already prepared for it. (Or rather: we think that we are, and reality will validate it.)

But the condition system has served its purpose by then! It helped us keep the system alive and fix the issue on the spot.

EDIT: The main issue that makes writing examples for this troublesome is the curse of knowledge. If I write that a condition system allows to recover from a type error, it's trivial to say, "just add a typecheck for that with a conversion routine"; if I write that a condition system allows you to recover from the format mismatch, it's trivial to say "but you can just stick an ifIsOldFormat(...) in the code to solve it". These solutions are no longer about exception handling; they are modifications to the standard control flow of the program. They aren't trying to fix the situation from outside; they are trying to fix the situation from the inside. And fixing it from inside isn't always viable or even possible at all.

While control flow deals with known knowns and known unknowns, exception handling is much more about situations where we deal with unknown unknowns: something explodes, we don't know what it is, none of our programmed recovery mechanisms have taken care of that situation, and we need means of gracefully recovering.

This is achieved first and foremost by allowing human intervention: the debugger is started, the stack information is preserved, so a programmer has all the information to try and debug the situation to let the program continue execution. Some of these fixes can then be easily added to code in by means of defining new programmatic condition handlers (hooks) and restarts (choices) in code.

(EDIT2: I wholeheartedly apologize for the wall of text. I'll just add it to the book instead - thank you for the good questions.)

> I'll claim, though, that if one is capable of computing these situations up front and preparing the recovery strategies ahead of time, then we're no longer doing exception handling whatsoever. If you are able to compute all situations up front, then you don't have exceptional situations, so you don't have exceptions and therefore don't need need exception handling!

Exactly! This is a cultural difference; in my world you should only use exceptions for things that should never happen, but I suppose 'never' is a matter of expectation and therefore code that follows that expectation.

> This is why one may also decide to solve this example by using a strong type system

Agreed, and to me this is the 'correct' solution. But that may be a counsel of perfection (and an overbearing cultural 'arrogance' on my part)

> That doesn't need to always be the case. Let's suppose that we've just run into the error and need to fix it - the situation that you described, an interactive hotfix of a running system.

This is both cultural and technical difference that I've been struggling with. This makes it clear what you're getting at.

> If you can successfully use the "parse, don't validate" pattern[...]

Not heard of this. You've given me some reading to do, thanks. For those unfamiliar, https://html.duckduckgo.com/html?q=parse%2C%20don%27t%20vali...

> We likely might need to develop a patch for that module where we now know that it has a fault and we may also need to live with it for X days before it is fixed upstream. When we have error-handling information stored in the dynamic environment and restart points sprayed across our stack, we have a situation where the error can be handled both without recompiling that faulting module and without destroying the stack - computation can continue when we've provided a proper recovery routine.

That is a compelling use case.

> We aren't getting crashes, stacks are not getting completely unwound.

I came across Lisper's enthusiasm for such exception handling ability (based on having the stack not-unwound) when I was trying to learn Dylan, a long time ago. I'm starting to see what they were getting at. That doesn't mean I'm comfortable with it, but I do understand it rather better now.

> along with the fact that Lisp is an interactive system (a lot of Lisp's power simply disappears when it's used like Java or Rust or any other batch language) [...] We can fix it interactively, and possibly store this fix in a new condition handler that we then patch on top of our running program [...] But the condition system has served its purpose by then! It helped us keep the system alive and fix the issue on the spot [...] These solutions are no longer about exception handling; they are modifications to the standard control flow of the program. They aren't trying to fix the situation from outside; they are trying to fix the situation from the inside. And fixing it from inside isn't always viable or even possible at all [...] This is achieved first and foremost by allowing human intervention: the debugger is started, the stack information is preserved, so a programmer has all the information to try and debug the situation to let the program continue execution

OK, these explicit statements are very important and revealing.

> ...apologize for the wall of text

No need, it's an excellent answer, it's just a perspective that does not come from 'my side of the tracks'.

Glad I persisted, I feel I've learnt something important today. Good luck with the book!

How should I credit you in the book's Hall of Fame?

Only if it's presented as a "motivating example", rather than "illustration".

  This is the TXR Lisp interactive listener of TXR 240.
  Quit with :quit or Ctrl-D on empty line. Ctrl-X ? for cheatsheet.
  1> (defun filter-using-exceptions(list)
       (build  ;; build macro: implicit, procedural list building
         (each ((item list))
            (catch (throw 'keep-it? item)
                   (yes () (add item))
                   (no ())))))
  filter-using-exceptions
  2> (handle (filter-using-exceptions (range 1 10))
             (keep-it? (item) (if (evenp item) (throw 'yes) (throw 'no))))
  (2 4 6 8 10)

  3> (defun filter-using-exceptions (list)
       (build
         (each ((item list))
            (catch (throw 'keep-it? item)
                   (yes () (add item))
                   (no ())
                   (substitute (this) (add this))))))
  filter-using-exceptions
  4> (handle (filter-using-exceptions (range 1 10))
             (keep-it? (item)
               (cond
                  ((< item 5) (throw 'yes))
                  ((= item 5) (throw 'substitute 42))
                  (t (throw 'no)))))
  (1 2 3 4 42)

In TXR, I simplified the concept by eliminating the duality of conditions and restarts. There are only exceptions. catch frames take an exception with unwinding. handle frames take an exception without unwinding (they can potentially decline it after looking at it). In the above example, keep-it? is an exception and so are yes, no and substitute.

So, why would you do the above instead of just, say:

  5> (mappend (lambda (item)
                (cond
                    ((< item 5) (list item))
                    ((= item 5) (list 42))
                    (t nil)))
              (range 1 10))
  (1 2 3 4 42)

In the exception example, that is happening also: a function is required from the caller to complete the behavior. But that function isn't passed as a parameter; it is dynamically bound as a handler which is labeled by an exception symbol. The function doesn't simply return a value, but chooses from a menu of restart points.

That points to a whole different set of ways of organizing the code, and options for extending and maintaining it.