I set up Sentry a few years ago, based on a PyCon hallway-track talk I saw. We had a few false starts with it, but have integrated it with a couple of our newer platforms and have liked it.

It takes a kind of "ticket" approach to messages, it'll deduplicate and combine similar errors, and you go into a dashboard and see "We got ten thousand of this error, let me track it down, fix it, ack it and see if we keep getting it."