I did SRE at another firm, not Google like the parent but a similar level. We tested these procedures twice a year and every time we added a region.
I agree: no excuse.
SBB just finished their project to be able to restart the grid in 5min. (There was a famous multi hour outage a few decade ago that I think motivated this work)
If the outage involves damage to transformers then all bets are off. In those cases some areas of the country might become uninhabitable for months.
A power grid is different because it's not about the power immediately being used all at once, it's about stabilizing the frequency. Turning generators on takes a while to stabilize, each reactor may output inconsistently for a while, which is why you never start a generator and immediately ask for full output, you start it, let it warm up, then add load. Which does sound like a server, but you don't need every server speaking the same frequency.
If a grids frequency is off, or stutters it kills electronics. If a servers JIT cache is ready, it's slow. The extremes are completely different.
Now the same garden hose problem exists, going from 0 to everyone logging in at once is a lot of load. But that's why you come back online with more servers and scale back.
It went remarkably well.
The initial start up (of the cold start plants) took the double of time expected. The "cold start" capacity is being increased due to some of the issues found.
Portugal was able to recover in 12h (still a lot) without any external assistance.
> By 07:00 CEST (05:00 UTC) on 29 April, electricity had been restored to 99% of energy demand, and full restoration was achieved by 11:00 CEST (09:00 UTC) that same day.
The wikipedia references seem valid for this precise timeline, which is more informative than your remark.
https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...
I'm not trying to win a pissing match, just trying to counter the very common miserabilist notion that Portugal did worse than Spain.
This specific problem was (1) caused in Spain, (2) fixed faster in Portugal, and (3) without external help.
Portugal could've been the cause, and could've done better, for sure.
But it's just not balanced to write "In Portugal it didn't go that well" as a response to "It went remarkably well [in Spain]." Makes it sound much worse than it actually was.
Do you have any engineering insight on why a hydropower plant was used, and not, say, a gas-fired power plant?
Once the hydro station is back online the power it generates can be used to provide house load to the other stations.
But load management while bootstrapping the rest of the grid is still very hard. Often the reason for really long blackout, ones measured in days rather than hours, is repeated cascading failures as operators try to reconnect parts of a the grid, and accidentally overload recovered parts of the grid, causing generators to trip off again.
Typically <5% afaik
They should have that and they typically pay substantially more for it. Unfortunately it's also the case that the private companies providing said black-start capability may have just not bothered to tell anyone they haven't been able to do it for a few months\years.
A properly run network will require those facilities to be tested multiple times a year to ensure it actually works, and make payment for the facility contingent on passing those tests.
So you need a hydro plant that is either small enough that it’s using turbines with at least some permanent magnets (you get turbine varieties that basically have a mini turbine inside using magnets to power the energise the larger electromagnets), or has dedicated infra to produce power to energise the coils in their turbines before they’re producing enough power to energise themselves.
The process is called islanding.
This is very much like trying to jump onto a train from another train moving alongside it.
We had to practice load pickup on a normal start at the pier often, but we likely practiced a full SCRAM to back online (including a FULL load shutdown) more.
Having predefined lists of what gets brought up at various points helps a ton, but it's still not an easy or simple operation.
You need a temporary source of power to start the water pumps and it takes a while for the water to flow all the way through the system and for the coal to saturate all the belts and power plants, if you hook it up to the main grid before all the plants are operational then it shuts down again.
Factorio makes this fairly easy for the most part, a cold start isn't really some massive engineering challenge, but still, worth guarding against. By the time you get solar and / or nuclear power is basically solved, at least on Nauvis, but it's "overengineering the game," so... we overengineer.
In Factorio, solar is an easy source of cold start. In the real world, I think hydropower is a commonly designated choice and the solar farms have to wait for a grid to sync against.
Cold starts are a constant in Aquilo until you research and build fusion power. And whatever you do by hand to handle them, there's a device in the game that does it automatically too.
Once we’ve restored power to the substations, we can carry out what are known as voltage restoration operations.
At a nuclear power station, if all goes well, there will be at least one nuclear reactor unit that has managed to operate in island mode. They have their own sub station and can restart the others units.
We restart as many units as possible on a virtually empty section of the grid (serving only essential and vital consumers such as hospitals). We then reconnect consumers in a predefined order to avoid significant fluctuations that could trigger further cascading failures.
The most difficult part is often identifying the exact cause of the blackout with certainty, so as to prevent it from repeating it.
“Ready? 10 seconds to power on server alpha-romero-one-o-four!”
“Load—at five percent, check! All systems stable!”
It's chaotic and stressful in a slow asynchronous way.
"Rabbit, flu shot, someone talk to me!"
(I have no experience with H1Bs nor beef, I'm a German who's merely had the joy of seeing failed offshore projects.)
All of it is documented through a unified catalog of services and their associated metadata.
Source: Worked in MSR and shipped services through Azure / CoreAI.
I mean... you can try, but unless you spend a lot of resources on it, I don't expect test load to be close enough to really practice a black start.
From experience, the best way to prepare for restoration from nasty outages is to have them frequently! The team I was on was pretty good at outage restoration, until we started regularly hitting about a year between outages. After a few years of one or maybe two outages per year, we had lost the muscle memory for protecting services that get overloaded during or after an outage and setting outage messages (which both informs customers and protects customer service queues).
Regularly starting up an isolated cluster at least lets you get ahead of sequencing issues though; hopefully it's enough if you can get the whole cluster started and then modulate inbound traffic (most complex services will need something like you mentioned so that you limit the number of sessions, but sessions that start are likely to proceed successfully).
This is one of those beautiful engineer understatements. Rotational speed is fundamentally linked to voltage by the motor velocity constant: https://en.wikipedia.org/wiki/Motor_constants
Although it's fundamental for all motors, the actual constant is arbitrary for each motor. Voltage is still proportional to change in field is proportional to speed; at zero voltage you have zero speed.
A large dead AC load has its own speed- zero. It's connecting the engine without a clutch, but the engine makes hundreds of megawatts. The 6' wide generator wants to go from 3600 RPM to a standstill in a few inches. A full speed rotor can have up to 5 gigajoules of rotational inertia; over a ton of TNT.
On a live grid you've got a synchronoscope: https://youtu.be/LVozbX9Mn1c?si=RhRPIkuTVxh4Wseq&t=181
Which is great, because connecting it backwards would cause a slow motion explosion. On a dead grid the only thing saving you is the resistance in the wires; it would be LOUD if you connected to it and you would very likely increase the ventilation through the walls of the plant.
That we often bypass local field control and tie large industrial synchronous and asynchronous (induction) motors directly to grid is narrow and quirky special case that is common for historic raisins but really quite awful, with staged delta-star or dahlander winding relay tricks to go from insane start currents down to just absurd.
Brushed motors are just synchronous motors with a primitive mechanical field control attached, allowing them to reduce the field rate in response to load.
How long do you need to dry a grape for it to not only become a raisin but a historic one?
In a synchronised power grid, machine speed is connected to electrical frequency, that's it.
Voltage and reactive power are controlled by the strength of the rotating field, not speed.
(It might be better not to use CIDR ranges, but use the lower bits instead if possible. For example starting to unblock all IPs that end with .80 until .90, to get more random geographical distribution)
Which leads to my suspicion it was internal load that brought the system down. Communication between the servers (like delivering queued up messages from one O365 account to another), and not external access from users and apps.
If the load balancer already does some connection tracking, it can just keep serving existing connections and only drop new ones. If connection tracking is not available, the IP blocklist would be a suitable option.
IP blocking is generally not what you want though.
Okay, they get different IPs during the day, fine. But it's still kind of randomized. Stepping through IP ranges will reduce load and give access to more people.
I really don't get the issue you are seeing with this.
IIRC Microsoft posted a comment around 10pm where they discussed rate-limiting external connections, and many reactions on last week's HN thread were like "wait, you so didn't even drop load before now?" So internal load may have been a contributing factor, but it's by no means certain given that they didn't stop serving clients.
This is absolutely terrifying knowledge to know given the competency of the current administration and their attitude/response to recent "black swan" outages.
Readers who know about distributed systems and their failure modes already understand the thing being explained. They (and you) are not the target audience. Let people who are not already experts learn something by analogy.
But the analogy is useful in another way. The electricity industry plans and rehearses for this kind of thing, even though it's impossible to simulate. I used to be rather skeptical that these rehearsals would be effective but Portugal black started last year after a solar plant in Spain malfunctioned and took down the whole Iberian grid. It took almost exactly 24 hours despite the grid collapsing several times during the process, which is the usual estimate for how long things will take. So it seems they have this stuff well under control.
I won't slam Microsoft too hard for this outage because this type of scenario has played out at many companies over the years. Yet I never encountered an ops team that has written procedures for how to recover when the entire service has entered a crash loop under peak load.
The exact analogy we use for this doesn't matter much as they are all pretty evocative, but using black start has the advantage of reminding us that there's another branch of engineering where this necessity is taken seriously and rehearsed. They also engineer for it ahead of time so they can prioritize customers like hospitals and airports under overload conditions. I think it would be a good feature for cloud LBs to be routinely differentiating between e.g. customers on paid plans vs free so they can serve paid customers first if there aren't enough live backends.
this is an interesting observation, since this should be table stakes for running a high scale distributed system. there are plenty of companies that do this and write about this extensively. it definitely becomes less important until you get to certain levels of scale though - a random app with 10k users has a different standard than, e.g., google cloud storage.
At the time Iberia was relying >50% on renewables which is very dangerous. The operators tried to recall gas power plants to provide inertial stability, but the plants were cold as they'd been taken offline in the morning to make way for solar (prioritized due to green politics), and even though gas plants can spin up very fast compared to other kinds they could not warm up fast enough to rescue the situation as it was all over thirty seconds later.
So it depends how you define "fault" but yes the starting point was a broken solar plant. Thermal/hydro plants stay locked together through physics because there's a prime mover connected to a generator, but solar plants rely on computer-managed simulations of the rotation that tries to follow the rest of the grid. When you have more simulation than reality it can get tricky fast.
Less neutrally, there have been far too many years of people pretending that you can build a grid out of 100% renewables. It was a false claim from the beginning, but engineering was overridden by ideology. The Iberian outage was one of the consequences - unfortunately merely a preview of coming attractions, as our political class is in denial and would rather believe lies than address the truth that it's caused by their own laws. Now it's only a matter of time until the entire UK or European synchronized grids experience a collapse of their own, with a similar root cause. We can't say we weren't warned.
For instance, a generator failure that cannot be mitigated by dropping load will probably result in islanding the grid: circuit breakers trip some lines and your network becomes several sub-networks. The sub-network that tripped can then be re-powered without a black start.
An important part is that clients pre-fail most requests and don't send them if the server is overloaded. They just assume the server would return an overload error.
The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.
This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.
Black starting a service is obviously hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I used to be an SRE at Google and although we regularly started up new clusters and moved traffic around between them, actually starting an isolated service instance entirely from scratch was never something we practiced. I remember in a few cases teams discovered devs had introduced circular dependencies by accident, i.e. starting service X required service X to already exist, or it depended on service Y that in turn depended on X. Sometimes you can only find these things by practicing.