Shortly before I joined in 2009, Google had embarked on a project to rewrite Search from the ground up. It was canceled after a couple years, and in many cases the work of individual subteams was folded into the main search engine.

Search also periodically rewrites parts of the search stack. Caffeine [1] was a complete rewrite of the indexing system, Hummingbird [2] is a near-complete rewrite of the ranking algorithm, Panda [3] and Penguin [4] were partial rewrites of the ranking algorithm, the serving system continually gets rewritten, and we redid the visual design every year starting in 2010 [5][6][7][8]. You'll notice that there was no 2012 redesign; it was canceled. That's what happens when the "new" search isn't better than the "old" search.

[1] http://googleblog.blogspot.com/2010/06/our-new-search-index-...

[2] https://en.wikipedia.org/wiki/Google_Hummingbird

[3] https://en.wikipedia.org/wiki/Google_Panda

[4] https://en.wikipedia.org/wiki/Google_Penguin

[5] http://googleblog.blogspot.com/2010/05/spring-metamorphosis-...

[6] http://www.theverge.com/2013/1/24/3904134/google-redesign-ho...

[7] http://searchengineland.com/google-launches-new-unified-desi...

[8] https://www.google.com/design/spec/material-design/introduct...

Can you disclose how much "better" for such rewrites? And what metrics were measured? I guess there is a conflict between CTRs on organic results and ads, so one has to be picked (or maybe a combo of the two). And what the magnitudes of the improvements? a few percentage or a few bp?

I can't disclose which metrics are measured, sorry. Basically, they had to be at least metrics-neutral with strategic value (where "strategic value" means "some exec thinks this is a good idea and we really need to do it"), or pretty strongly metrics-positive without an exec's personal interest. Conflicts between CTRs on organic results vs. ads happen less often than you think, they're usually an issue only on full-page redesigns. I was an informed bystander for a few of them - I can't detail the precise process, but let's say that it's a complex balancing act that tries to balance the interests of all constituencies: users, webmasters, advertisers, and shareholders. (The one constituency that doesn't get a break are the poor engineers and PMs that have to implement it all..we often had some incredibly complex solutions to make the page marginally more user-friendly.)