What About the Jobs?
With more and more of the company’s data centers shifting to automated infrastructure control, and with the real possibility that the same will eventually start happening outside of Google, arises the inevitable question of jobs. Are Google’s data center engineers engineering themselves and their colleagues out of work?
So far, Kava hasn’t seen evidence of that happening.
“We still have people there, because they still have to do all the maintenance,” he said. “So, you’re not getting rid of the people, you’re augmenting” the existing team’s capabilities. “Instead of trying to tune the system themselves, they can focus more of their time on preventative maintenance and corrective repairs.”
Besides, AI still does poorly in situations “outside of the envelope of its training,” he said. People are very good at making observations in what Kava likes to call “corner cases” and coming up with a course of action on the spot. AI isn’t.
In other words, it’s a good idea to have AI fine-tune a cooling system to improve efficiency in pre-tornado conditions, but you better have some human engineers around in case a tornado forms.
I’m sure early automakers “saw no evidence” in jobs for carriage drivers too.
I guess the solution is to simulate faults everywhere at all times in production. Something most systems definitely ignore. Fault tolerance is just too different from "normal" way of writing software, it can't be done well without paradigm shift.
I mean imagine if you have a distributed system like an object storage or a database. Before deploying it into production you try to evaluate how it performs in various conditions. But you can't really do it thoroughly enough and on a lot of data, you can only scratch the surface and this forms your expectations about performance. You deploy it into production and start monitoring how it performs and relying on that performance that you barely understand. Everything seems fine, until you add nodes to the system or replace an hdd and suddenly it becomes so slow, that nothing works anymore. Turns out everyone, including system designers, had broken assumptions about its performance and didn't even think about it, because testing fault tolerance was always done only separately from normal mode of operation and it was impossible to grasp how they affect each other and come up with better algorithms.
> Automatic Failover to a neutral state if the AI control system does violate safety constraints. Smooth Transfer during failovers to prevent sudden changes to the system. Rules and heuristics as backup if we need to exit AI control mode.
How long until management demands the datacenters work with cooling that just provides, say, five nines of cooling capacity ? Can you guarantee that doesn't happen ? Do engineers at Google even care that that doesn't happen ?
In other words, how long until Google loses the ability to pull the plug on this system from a physical perspective ? How long until Google loses that ability from an economic perspective (that turning it off would incur unacceptable costs) ?
And if it ever reaches the point where the AI system is so reliable that they're comfortable ditching the backups, I'd call that a win.
I ask because I've implemented systems that allowed operators of large networks to do exponentially more work, and therefore grow the network while keeping operators down. It essentially reacted automatically to random outages, tracking their status and carefully bringing things back up when possible. Then the system went down, and we started repairing the system under the knowledge that we had to call up everyone extra to do the (very tedious) work that this system did. And for those of use repairing the system we were working against the clock: we knew we had about 2 hours before we'd start missing contractual obligations to customers, and about 2 days before we were likely to experience serious degradation in network performance and bandwidth, and this couldn't be fixed, as in it would happen regardless of any realistic amount of human effort invested in manually doing this.
We fixed it (in about 1h15m) but I'd be lying if I said the patch was properly tested when we set it live.
The scary thing is, this happened ~8 years ago, and now we've pretty much had a "full turnaround" in the "manual" team: only 2 (out of ~80) have ever manually taken care of these actions. If it were to fail now, and those 2 are on vacation ... hell it's been working so well I wonder if they'd even notice before it's too late.
So what happens to the DCs when the AI disconnects somehow from the equipment ? Shutdown ?