Ugh this happens to me all the time, but on the other hand I'll sometimes be scrolling through a pdf with my trackpad, but typing in a pane on the left, so I do find it quite useful to have that split between kb/pointer focus.