>event listener that fires on every mousemove event to pan the canvas... it's not the most expensive calculation, but it fires quite a bit, causing the browser to constantly spend some time scripting and re-rerendering.

Look at "Throttling and Debouncing Events", you don't need to process every single event as humans are not that accurate. It will make you application perceived much more smoothly.

Article about the concept.

https://www.digitalocean.com/community/tutorials/vuejs-lodas...

Next point to consider is using web worker thread for calculation and not blocking main thread. Another article about the concept,

https://medium.com/launch-school/what-are-web-workers-4a0e1d...

Have fun