It's the duration that gets you. For example, if you go out in a 3/2 (3mm torso, 2mm arms/legs) when the water temps call for a 4/3, sure, you can surf for 45 minutes or so, but after that you'll start getting quite chilly.

This guy spent 75 hours floating in the water, not paddling around expending tons of energy competing with the pack for the next set wave :)