Look into HP patents around 1990-99 on optical paper sensors. Afaik mouse sensor came from research into large format plotter paper tracking.

US Patent 5,729,008. Interview with inventor: Oral History of Travis Blalock https://www.youtube.com/watch?v=wmqa9XJED-Q https://archive.computerhistory.org/resources/access/text/20...

"each array element had nearest neighbor connectivity so you would calculate nine correlations, an autocorrelation and eight cross-correlations, with each of your eight nearest neighbors, the diagonals and the perpendicular, and then you could interpolate in correlation space where the best fit was. "

all in analog domain

another one from HP/Agilent/Avago: Oral History of Gary Gordon https://www.youtube.com/watch?v=TxxoWhCzIeU

also this https://blueironip.com/hp-capshare-how-a-failed-product-yiel...

A USB mouse just sends x/y distance traveled over USB HID protocol. All the details are integrated into silicon. The sensor is a camera and the processor diffs the images to calculate the distance. To learn about the algorithms for that you could look into patents by Avago or PixArt.