Weird, I never saw it as a punishment. It's actually a lot faster to get where you need to go, in my experience. Not having to filter through both application and window makes it take slightly less thought and effort than flipping through just one application's windows. Plus switching between applications is a lot easier when there are only five listed instead of five + 20 windows.

I guess it's dependent on the number of applications and windows you have going at any given time.