I do it every day and never have issues with scrolling vs pasting.

The biggest issue is that some apps don't play nice with the x clipboard.

When I hold down the middle mouse button to scroll, change my mind, and then release the middle mouse button to move the cursor elsewhere, that results in an unwanted paste. That has happened enough times, especially in code editors, for the selection clipboard to be an annoyance for me whenever I am using a keyboard equipped with a pointing stick.

Some desktop environments and Linux apps have the option to turn off middle-click paste, though this preference is not always honored. I respect the preference of those who do like the selection keyboard, and I don't even have a problem with it being the default. But, I don't think it should be forced on all users, and I don't think decisions about other core desktop functions (like Ctrl+C and Ctrl+V) should be made under the assumption that the selection clipboard is enabled.

> That has happened enough times, especially in code editors

Ah, that's probably the difference. I don't usually scroll through a text editor (as I'm usually using vim). I can definitely see how it'd be an issue in a graphical text editor.