Same for Start and End, really, if you adopt the 'show the last page or two' approach, so this is possibly better:
[1] [2] 3 [4] [5] ... [19] [20]
You can then omit a whole class of concerns and just concentrate on the few remaining elements relating to the page numbers themselves.Also, you're forgetting about the fact the bottom of the page forms a boundary. If you're pagination is at the bottom of the page it is consistently the same distance from the bottom after you scroll all the way down.
If you have a mousewheel you can take advantage of this on Google. After you click next the first time, leave your cursor in the same position and use the mousewheel to scroll as you scan the results. Once you hit the bottom, the cursor is over the next link and you can just click.
Even if the user isn't advanced enough to use a mousewheel, they can still become familiar with the position of the next link relative to bottom left of the page.
Took me a while to understand that "Next" works differently and is somewhat misplaced after "10".