The real world is too complex to analyse. Theory works by analysing a simplified model of the world, with the hope that essential characteristics are maintained and only irrelevant details are ignored.

Take the analysis of algorithms. I expect most of the analysis you've seen assumes memory access takes constant time. This hasn't been the case for a long time, due to caching. Now most people don't really care about performance so it's a reasonable simplification. However some people really care about performance and thus need a more complex model. See cache-oblivious algorithms for some work that addresses this. When you get to the level of assembly language optimisations no model is going to help you. You just need to benchmark the code on a real system.

Similarly in distributed systems we work from simplified models. This is fine for most of us, but Google has one of the most complex systems in the world. Naturally their requirements are a bit off the beaten track, and perhaps not adequately addressed by theory.

Yes, and often times theory consciously sweeps essential things under the rug. Take for example physics which postulates frictionless surfaces. That must've seemed like a ridiculous loss of explanatory power. But it turns out that humans can get principles that way, if we blatantly take a huge amount of reality and dismiss it as "friction".

Science really is a different enterprise than engineering. I hear it's fairly recent that scientists had anything to say to engineers and those in medicine. (Now of course science is vital to these enterprises.)

Decentralization is Hard. Really hard. It brings in all kinds of fun issues - What if you get bit flips on one particular node? What if a random host fails? What if your load on your hosts is not perfectly balanced? What happens when you're halfway through a software deployment and half of your system's running a different version of the software? Not to mention most people aren't great at reasoning about even just multiple fault-free threads in the same address space, much less multiple threads on multiple machines facing potential byzantine failure conditions.

Centralizing bits of it therefore will make these issues easier - at the expense of reduced scalability. So then the question becomes, "Can I get away with centralization here?" You look at the performance and failover characteristics required, and see if centralization will work for you over the next N years, and see what your plan (and timeline) for scaling out when your central box isn't enough would look like. And if you find you can get away with it, you probably should.

Of course, nothing ever works out quite the way you expected. Maybe your fancy algorithm requires more memory than you thought. Maybe failover's a bit too slow. Maybe it's eight times more CPU-intensive than you expected and that one box isn't going to be enough. Things happen, and sometimes you get it wrong. Still, starting from a default position of "let's centralize what we can" will save you a lot of headaches, just because it's so much easier to reason about the system's behavior that way.

It's hard to measure complexity and predict network usage. Centralized systems are also a lot easier to design and implement and make fast. Making good decentralized systems is a real research project, and often flat-out impossible. Often, decentralized systems push down their complexity to the user of the system -- see the decentralized key-value stores that push conflict handling down to the client.

human behaviour and the environment can't be modelled. Real traffic is bursty. Analytical tools have to parametrize noise into unrealistic distributions like Gaussian, Poisson in order to get an answer. Real data has lots of hidden dependencies which is hard to model. An earthquake might make everybody in a geographic area use the internet at the same time. That kind of cross correlation is hard to model correctly. The only sure way of knowing a system behaves well in a realistic situation, is to run it. Real data can answer questions like how big the caches need to be etc.

Your question is essentially "why is there a difference between theory and practice?".

in theory there aren't

One reason is that modern services aren't just a single algorithm. Large scale services do exactly what you're talking about, but each feature of a service usually maps to many algorithms, each with different characteristics.

Also, don't mix up control and the actual data/processing. Google is using centralized control, but the data storage and processing is still distributed.

I think this boils down to two things: We're not that smart and the problems we're solving are either too ambiguous or have too many engineering dimensions.

A lot of the people implementing distributed systems have some knowledge of CAP, FLP and know of solutions to those problems- but compared to dist. sys. researchers it's somewhat superficial. I also think there is a bit of posturing in CS; people are afraid to not grok these kinds of systems and so instead go off to build the wrong thing.

You can't tell how software/architecture built today will be used in the future. You do your best to pick a method, but you operate on imperfect information. Sometimes you get it right and things work well, sometimes you get it wrong, and things still work well. Sometimes you get it right, but things change and it no longer works well, and sometimes you get it wrong, but things change and it becomes a good solution.