Layman’s opinion but two perspectives on this from physical motion might be interesting.

I guess the position of the mouse is determined by wrist joint plus fingers and many muscles in all of these act together to perform fast and precise movements. Controllers typically feature sticks each controlled by only one finger and it’s muscles. The fewer muscles involved could mean fewer options for combined complex/fast movements.

Also with a mouse a larger physical plane (some houndreds of square centimeters) in real space is translated into virtual space. With a controller the movement inside a very few centimeters is relevant and the smaller movements simply are not as precise as larger ones when projected into the same virtual space.