The only size-related optimization that I can think of is that V8 doesn't inline functions (but does optimize them) when they are more than X characters long, comments included.
I'm not sure about SpiderMonkey, but they may very well use this kind of optimization too. However, I would guess that Uglifying files should not increase execution speed for any noticeable amount.
I'm not aware of any common minification techniques that hurt performance. They generally improve parser performance and reduce memory usage (less memory spent on line-ending data and script sources, for example)
It's a good question but minification shouldn't cause runtime performance issues. Keep in mind that these are mostly micro optimizations when running outside a large loop. If you go from 1 mil ops per sec to 20 mil ops per sec your user won't see a difference unless the code is being run many times over.
Is it probable that SpiderMonkey and other engines suffer from forms of this deoptimization hell when rendering? And would it be a safe guess that whatever mod_pagespeed does to a page's javascript resources is less likely to result in this hell than these other tools? Thanks.