I can't explain exactly why it is annoying as currently implemented, but it is. Somehow, the more results thing needs to be more "fluid". Maybe make the interface more like regular pagination, but just add the results to the bottom of the same page?
1) When you click 'more results' add a divider before the first new result to make it clear which new results just appeared. In most cases the first new link will already be highlighted since the mouse will be hovering overy it but any antsy mouser will not have this benefit.
Also, I notice that I have the tendency (and need) to move the mouse off the first new link so I can read it. This is very inefficient so it would best to append the new results one line below the 'more links' link.
2) Consider adding an auto-scroll to put the first new result at the top or middle of the page to reduce scrolling. This may be disruptive, however.
Seems to give some good results back in a useful way.