Google historically splits off their "front end" which is dealing with messages from the internet and their "back end" which is processing what those messages want to do. It was briefly described in the original search paper they presented.

Of necessity, most machines that Google runs have 'private' (as in not directly addressable) addresses. This is how it seems that everyone runs things when they get above a certain number of servers.

Thirdly because one can assume that 'attacks' (whether they depend on XSS, overflows, or what not) are coming from the Interwebz its prudent to present the smallest possible attack surface on a machine which is being fed internet stuff. As you can imagine, those security constraints make for uncomfortable restrictions on what you can do in the 'front end', so a machine that is 'front end' has a different set of security constraints than a machine that is 'back end' (they are still constraints there of course but having gone through the filter of the front end machine which has greatly attenuated the possible exploit vectors).

I believe it is this armored 'front end' resource which is the scarce resource, but again, I've been gone over a year and things change quickly inside the 'plex.

Not trying to piss all over this, since I thought you might have been saying something I didn't know. But I have worked at Google for a long time and basically everything you wrote is incorrect and has been since at least 2005 before App Engine even existed (why would a paper written in 1998 be relevant?).

Without going into too much detail, it shouldn't surprise anyone that physical machines are interchangeable via cluster management software, so application front end and back end software all end up running on the same machines. They aren't addressed through DNS but by another mechanism. There is no special system administration for application-specific front ends vs back ends. Spinning up more instances of a particular server is a 1 line change. Reverse proxies (i.e. a non-app-specific front end) are also involved but they're not a bottleneck AFAIK (this problem is well "commoditized" in open source whereas other parts of Google's stack are not).

I would be curious to see a comparison with competitors like Heroku, which I've heard are pretty expensive too. Theoretically Google should be cheaper because Heroku runs on AWS and thus is paying for Amazon's profits (I have no idea how they compare.)

The original point is "almost right". The scarce resource reflected in the new pricing model is not machines, but RAM. The old pricing model assumed CPU time was the essential resource. The new pricing model bills you for the amount of time your instance (which has a fixed RAM reservation) stays resident.

It sounds plausible that "on demand" instances i.e. search servers are a bottleneck, while bulk offline processing (in the back-end) i.e. PageRank calculations is not such a constraint.

Yes, those offline processing instances could be suspended while dealing with a traffic peak, but peak traffic is still often a challenge, especially now that Google is pushing more and more stuff onto the web servers (Instant).

That might be what he meant.

Disclaimer - I don't work for Google, it's just a guess.

I interpreted it more as saying that their architecture allows them to parallelize all sorts of 'back end computations' across their infrastructure, such that their scarcity is in internet-facing processing and not processing as a whole.

Probably he is saying that because their expertise and focus lies in backend, it's costlier for Google to increase frontend service and server compared with say, other web-hosting companies.

Web servers still count as backend. This is not a lack of talent problem.

People who are downvoting me don't really understand what I'm saying. I'm finishing up my internship at Google now. This is a nomenclature problem -- what ChuckMcM is properly referring to is front-facing backend vs "back" facing backend. They're really just two different types of backend servers. SREs manage both and they are installed similarly. It could be argued that Google perhaps has certain weaknesses in frontend (i.e., UI/UX design), but this is clearly a case where engineering talent is not lacking -- they're the same engineers as the ones working on private servers.

I think I know what he's talking about. My LDAP is the same as the email in my profile. I'll wager a guess at what the real problem is.