Moving to the mouse is a movement where the distance from my hand to the body stays the same, while moving to the trackpad is a movement where I need to being my hand closer. For whatever reason, I find the second easier.

Additionally, when I'm using it on my lap, the trackpad is basically next to my body, which makes my wrist position a little uncomfortable.

Looking back, I think it's not really the switching, but rather the comfort when using one for a prolonged period, for which I found the trackball easier.