Moving to the mouse is a movement where the distance from my hand to the body stays the same, while moving to the trackpad is a movement where I need to being my hand closer. For whatever reason, I find the second easier.

Additionally, when I'm using it on my lap, the trackpad is basically next to my body, which makes my wrist position a little uncomfortable.

Looking back, I think it's not really the switching, but rather the comfort when using one for a prolonged period, for which I found the trackball easier.

The angle on the trackpad can be uncomfortable when it’s sitting on your lap.

I find that the larger trackpads on newer macbooks help with this a lot because the upper right corner of the pad is easily accessible at exactly the same angle of inward rotation by just pulling your elbow back along the side of your body. The Apple magic, I guess, is being able to have that without constant phantom palm touches.

See https://imgur.com/a/qeRQIFD

An offset trackpad would also work, though obviously would be less friendly to lefties.