Colemak all muscle memory for me, so it's nearly impossible for me to remember my Qwerty muscle memory now.

One place that really sticks out is job interviews. Asking your interviewer to change keyboard layouts is a bit intrusive. It's not a huge deal on a Mac, but it's a challenge on Windows since Colemak isn't built in.