I often find myself using "except Exception", while I don't like it, I cannot find another way.

For instance when making an HTTP request. The requests library can throw thousands of different errors (SSLError, BrokenPipe, socket errors, errors from urllib, errors from requests itself...).

As far as I know, it is the only choice if you want to know if a request succeeded or not so you can deal with it without your view returning a 500.

I would really appreciate if someone has another solution to this.

IMO, there's nothing wrong with "except Exception" if your handler is, in fact, meant to handle all exceptions.

(Well, there's one thing wrong, which is that certain standard library errors in Python 2.x don't derive from Exception [socket error, I'm looking at you]. This has been addressed in more recent pythons, but there's still lots of 2.x out there.)

Python's rules for handling exceptions in a class hierarchy are designed to provide programmers with the ability to handle errors at whatever level of granularity makes sense. If your requirements say that the software should treat SSLError and BrokenPipe differently, then by all means, handle them separately.

More often, however, you're making a library call for which any failure has identical requirements. When you're trying to read a configuration file, you probably don't care whether a hard drive is corrupted, has a loose power cable, is on fire, or somehow triggered a divide by zero error somewhere in the bowels of an I/O library.

  >>> import socket
  >>> issubclass(socket.error, Exception)
  True

It's been like this since Python 2.4 at least.

I stand corrected. Well, partly. Looks like it was changed in 2.6.

https://docs.python.org/2/library/socket.html#socket.error

All of requests' exceptions inherit from requests.exceptions.RequestException [0], so you could catch it.

[0]: http://docs.python-requests.org/en/master/_modules/requests/...

Kennethreitz requests lib? Just use requests.exceptions.RequestException, that should handle every error, if not then report it.

For python urllib IOError and OSError should suffice. At least in py3.

  import urllib.request
  try:
      urllib.request.urlopen('https://wrong.host.badssl.com/')
  except (IOError, OSError):
      pass

causes:

  ssl.CertificateError: hostname 'wrong.host.badssl.com' doesn't match either of '*.badssl.com', 'badssl.com'

and in Python >=3.3, IOError is an alias for OSError.

You should have (+/-) one "except Exception" in your code which would be the "catch all/log error" thing.

But for how to do with errors, see the sibling comments to this one, and good libraries will give you an inheritance tree of exceptions

This is basically the equivalent of returning a 500 or failing a task. You know that your code failed somewhere but there is no way you can recover from it (granted your code is not idempotent so retrying everything is not an option).

By catching all exceptions as close as possible of the request you make, you can at least do something about it.

As for the tree of exceptions, it works well for a while until networks become unreliable, until remote peers start presenting certificates broken in subtle ways... You end up adding ranges of exception in the except clause as new ones are popping on Sentry. Then you get tired of it so you catch "Exception".

This is the unfortunate story of every single project I do that deals with network calls on the Internet.

Where 'in your code' is a module. Then log the exception and raise a new exception with a module-specific code. requests.exceptions.RequestException is a good example.

My modules do catch SystemExit and KeyboardInterrupt before the bare exception and simply re-raise those.

Good unittesting can trigger the relevant exceptions. I know this can be a lot of work to write tests which trigger all your corner cases. I practiced this in a recent project but now I'm much more confident in my code.

Another strategy is to catch the "expected" exceptions first followed by a general exception handler. This can use the `traceback` module to log the call chain and error message.

A better way is to implement a logging function, which will be recording unknown exceptions the way you prefer: sending an email, writing to a text file, whatever. Then you just write:

    try:
        [...]
    except Exception as e:
        log(e)

It's not much longer than `except Exception: pass` and you gain more control over your system.

Requests cannot throw errors from urllib because Requests does not ever invoke urllib code.

Any error that does not inherit from our top-level exception is a bug: please let us know so we can fix it.

I am working on a crawler that uses async/await, and from experimentation the list is:

except (aiohttp.ClientError, aiohttp.DisconnectedError, aiohttp.HttpProcessingError, aiodns.error.DNSError, asyncio.TimeoutError, RuntimeError) as e:

I want to continue running no matter what, so I also have an "except Exception", but I have better logging now that I know what the known vs. unknown exceptions are.

My pattern on this stuff is to initially write:

  try:
      do_something()
  except ExpectedException:
      handle_it()
  except OtherKnownException:
      handle_that_too()
  except Exception as exc:
      log.exception('New kind of error just happened')
      handle_fallback()

Yes, that's basically what I'm doing.

>I often find myself using "except Exception", while I don't like it, I cannot find another way.

This (pokemon exception handling) is a massively, massively destructive anti-pattern.

Once a code base grows beyond a certain size it usually leads to failures which manifest themselves with strange behavior that takes hours (sometimes days) to track down the real source of.

    try:
        ...
    except (NameError, AttributeError, TypeError):
        # likely programming error
        raise
    except Exception:
        ...

Not pretty, but maybe better than catching every exception.

>There are cases where you could justify writing "except Exception as e" and having it return a value

I actually use this pattern quite frequently in one very specific place. I use marshmallow to handle schemas for my API. On endpoints that require creating or updating a model, it's often difficult to write an exception for every single possible error type. Therefore, I usually try to catch ValueErrors, IndexErrors, etc explicitly. However, as a last line of defense I do have an except Exception as e. This is to catch and document any exception that I may not know of. Ideally, this ends up catching library defined exceptions that do not inherit from a standard defined exception or inherit from Exception directly. I haven't hit this line yet on any of my views, but when I do I'll know exactly what exception I need to catch.

except Exception: is useful when you have two alternate ways of accomplishing a goal; one default that might fail, and one extra that is less complete. Although you might only expect a certain error, since you have an alternate, you might as well use it for all errors.