You say that you hardly ever use the mouse. If you’re out of practice then you’d probably be slow at it.
On a good trackpad you don’t actually need much visual feedback because the response is very predictable. You can make a short movement and click more quickly than you can press a couple of keys.
Saka Key looks to be both slower and more complex than navigating using a trackpad (especially once you take scrolling into account).
With a pointer you treat the screen as a continuous space. With Saka Key's link clicking method (for the lack of a better term) you're selecting out of a discreet set of keys. I'm not sure what the implications are, but I feel like the difference is qualitative.
I guess which is better for me doesn't have to be so for someone else. Mouse interfaces do have a certain simplicity to them.
Check demo animation on Saka Key's readme: https://github.com/lusakasa/saka-key