Python, AFAIK, doesnt have a 'debug' mode, so its likely the IDE that is slowing down your debugging experience. Im not sure how you work around that - some IDEs do a lot of superfluous stuff when 'debugging', and may have settings to toggle features.

You can also just import the debugger where you need to use it, in which case you shouldn't get any slowdown.