I just gave up trying to read a newsweek mobile form article because of this (site kept resetting the scrollpoint to the top of page). If I can't read a text article on my $500 pocket super computer, I'm beyond annoyed.
And I agree, even if your scroll .js works perfectly, the callbacks leave a perceptible pause on most phones.
Because I'm traveling I'm using a cheap 70$ Android. So the phone isnt very fast and the internet connections slow. This makes a good number of "modern", especally SPA news sites completely unusable. Scrolling callbacks constantly freezing the whole interface. It's not per se the fault of using too much javascript – gmail and facebook work pretty well. But if you dont have ressources to optimize your sites fancy js magic please consider not doing it at all.
I dont expect much of my cheap phone. But reading plain text is definitely something i do.