Well here's my issue with this explanation that I've heard a lot: the caller may or may not be able to recover from it, but the caller may still be doing something with the error. Maybe the caller doesn't want to expose the error to its caller, or maybe the caller wants to build a new message to explain the cause of your error. But to panic! takes away a lot of options for the caller.

Worst of all, it leads to untestable code, because the calling test case can't properly check the error against some known result.