That's not an idiom python has ever subscribed to though, it's always subscribed to the "Better to ask forgiveness than permission" - I think there are strengths to both viewpoints, but honestly I think "don't use exceptions for control flow" is more of a convention that a "truth"

In absolute terms or when coding on paper, perhaps. But in the real world and if performance even remotely matters, it’s as close to a universal rule all languages end up embracing or turning into creaking hulks of slow code given how exceptions work in practice at the level of cpu execution units.

C#/IL/.NET embraced excerpts heavily at around the same time (start of the 2000s) but in time developers (in and out of Microsoft) learned the hard way that it doesn’t scale. With .NET core, exceptions for flow control are completely verboten and APIs have been introduced to provide alternatives where missing. Exceptions should be so rare if you chart all handled exceptions, you shouldn’t see any and would thoroughly explore why one or more pop up when a system or dependency hasn’t exploded.

Exceptions have two core properties: (1) non-local jump (carrying a value) and (2) dynamic binding of place to jump to. Contrast this with e.g. break / continue in loops where (2) does not hold. If most use cases of performant OCaml exceptions were not making use of (2) that would be an interesting insight for programming language design.

Have you got data on this matter?

   let modify_opt a f l =
     let rec aux p = function
       | [] ->
         (match f None with
          | None   -> raise Exit
          | Some v -> rev ((a,v)::p))
       | (a',b)::t when a' = a ->
         (match f (Some b) with
          | None    -> rev_append p t
          | Some b' -> rev_append ((a,b')::p) t)
       | p'::t ->
         aux (p'::p) t
     in
     try aux [] l with Exit -> l

That's, as you say, a statically scoped fast exit. One does not need the full power of exceptions for this (exceptions'd dynamic binding comes with a cost). If exceptions are widely used for this purpose, one might consider adding a nicely structured goto to the language. Something like linear delimited continuations?

For instance, the basic stream/iterator in Ocaml is Enum, which always uses an exception to signal when the stream has been exhausted, rather than providing a "has_next" predicate.

The catch site of this exception is dynamic.

Having ported some decently large codebases from OCaml to F#, the heavy use of exceptions in OCaml (where exceptions are very lightweight by design) had to be changed to normal control flow with monads to achieve good performance in F# — specifically because of .NET’s slow exception handling.

Python explicitly does not agree with this...

The same for a lot of your other complaints. It sounds like you are trying to write some other language in Python. Similarly if someone in a team using Java tried writing Python in Java they would complain a lot and end up with ugly hard to work with Java.

I think StopIteration is being removed. That was a a bad idea.

In fact, the Python developers recently doubled-down on the design by introducing `StopAsyncIteration`.

PEP 479 is specifically related to generators:

> If a StopIteration is about to bubble out of a generator frame, it is replaced with RuntimeError, which causes the next() call (which invoked the generator) to fail, passing that exception out. From then on it's just like any old exception.

OK, I'm willing to contend that that is the case. It doesn't have much to do with my overall issue with error handling.

> The same for a lot of your other complaints. It sounds like you are trying to write some other language in Python. Similarly if someone in a team using Java tried writing Python in Java they would complain a lot and end up with ugly hard to work with Java.

This is a very loose criticism of my post, I don't know how to respond to this. I've written Python for years, I think I gave it due credit for what it's good at.

I am not trying to write some other language with Python, I just think Python is not a very good language compared to others given a lot of fairly typical constraints.

I've written Python for years, I think I gave it due credit for what it's good at.

I'm not sure how fair this assessment is, considering the multitude of Python projects of considerable size and scope.

It would be very silly for me to say that you can't build large systems in Python, I've worked on plenty that are much larger myself.

Saying a language is possibly the best for two important use cases (exploration, small programs) is quite a statement, in my opinion. I don't think I believe that there's a language that excels so well in, say, building web services.

Whats the convincing argument for defending this? Ive always heard its just the way python is. Through experience using exceptions for anything other then exceptional situations and errors seems messy.

Imagine you are working through a series of EDI files and trying to post them to a badly documented Rest(ish) API of some enterprise system. If the file is bad (for whatever reason) you need to log the exception and put the file in a bad directory.

Pythons use of exceptions for control flow is perfect for this. If file doesn't load for whatever undocumented reason, revert back to log and handle the fallout.

"Oh I see a pattern, this API doesn't like address lines over 40 characters, I will add a custom exception to make the logging clearer, and go and try and see if I can fix this upstream. If not I will have to write some validation"

It is this dirty world of taking some other systems dirty data and posting it to some other systems dirty API that I find Python rules.

I have never worked on a large application where I owned the data end-to-end. Maybe there are better choices than Python for that?

That said, the amount of cpu cycles wasted even with best of breed pandas is insane, and when you want to do something “pythonic” on big data it all falls down. When you want to deploy that model you are also going to have problems.

That said, it’s still the best tool for the job, but it’s certainly not because of the creators of Python.

Python without type annotations can be painful in an IDE - it chokes a lot on those features.

Since I don't take a very type driven approach to Python (it would be too slow since I'd have to figure out 3rd party library types and shit like that) I just write annotations in places where I personally know the type. Mypy complains about this for various reasons - probably because my annotations are not technically correct, because I'm not a pro at generics in mypy and working out inheritance, as well as general mypy issues like poor support for class methods and that sort of thing.

But I ignore all of those errors because the IDE can still work things out.

I see the point that python invites these anti patterns.

But on the other hand, a software developer that returns Union(list[item], item]) in Python is probably also going to mess up a Java program sooner or later.

Why is multiprocessing a dead end?

The overhead obviously is still there, but the interface is a drop in replacement for threadpoolexecutor, which looks basically just like a multithreaded/async/future-based collect.

To communicate across mp you use pickle, which is a pretty significant performance impact relative to something like threading. There's also the issue of copy on write memory + reference counting interacting poorly.

I suspect this is the root cause of the difference in our experiences. My uses of mp have usually been somewhat more "embarrassingly parallel", for instance having a list of data elements which need to be processed with the same algorithm. For this use case, the usage of mp is pretty simple, often only a `pool.map(f, xs)`.

I can imagine that pickle might have tricky edge cases and/or be slow.

a) All reference counts after the fork are going to cause a copy of memory, so any memory access (even a read) can trigger a copy.

b) Even to send 'xs' over you must serialize it via pickle, and then the callee must deserialize it

That may be fine for your use case but it's strictly worse than just sharing a reference across a thread.