The problem is that Firefox on Mac is a second class citizen, and it uses modal dialog boxes.

As for the Esc thing, I can't recall a case where a Mac native application didn't press "Cancel" when I hit escape. I think there might be problems with ported applications though.