As you will immediately notice FB, Twitter, Whitehouse.gov, barackobama.com are repeated 3-4 times each, and this is only the 1st 30 results you could continue scrolling down infinitely and you are bound to see the repeat again and again.
My point was user feedback complained of repeated results on my search engine. Users generally thought my results were lower quality than Bing/Google despite being identical depending on the API I was using. Further, I believe a lot of this has to do with Google/Bing users being unable to see a large number of results on the 1st page.
But even linking to different subpages I think Google would prefer users not see multiple FB, Wikipedia, ect... results and that is just one rationale I offer for Google having an interest in limiting results/page (the question of the original submission).
I created such a search engine myself, but instead of displaying the standard text results I displayed only the website favicons so as you scrolled down infinite favicons appear, click a favicon it takes you to the webpage.
I think nickporter hit the nail on the head about the first page of results being so important and as you mentioned the competition to be on Google's 1st page is good for Google financially. However, one complaint about my search engine was that since the results often repeated the same favicon users thought the results were subpar, never realizing the results were Google results all along. By limiting the results per page it is harder for people to see how often Google's search results repeat, which according to my feedback users associate with low quality results.