There is a story that the white house was actually burnt by some of the many former slaves who fought with the brits/canadians. It sounds too fanciful to be true, but freed slaves were there and would have participated in the burning/looting of DC.