I know a couple of Finns that have swum everyday for decades, no matter what the water temperature (eg through ice). They claim it builds up brown fat. Maybe the ratio of brown fat has something to do with it.

http://en.wikipedia.org/wiki/Brown_adipose_tissue