A line goes out. 
  while (there some lines are still live):
    The current redistributes itself among neighboring lines.
    Neighboring lines trip
  end
  BlackStart()

Usually the hydroelectric plants are started first. Arg. is lucky, they have a lot of those. Otherwise you need to fire up diesel generators (which hopefully have been maintained).

You give power to the grid first so you can start another plant.

Then you feed hospitals and continue firing the thermal plants.

Throughout all of this you have to keep phase stability (no grid, no phase reference). I'd assume this would limit renewable's use-fullness for re-start (a thousand 5Mw turbines out of phase are best left idling)

As a last step you turn on your nuclear plants (Arg. has a few). These are last because of nuclear poisons that accumulate from decaying wastes that weren't burned while the reactor was off.

It's a bloody hard problem.

I forget the exact calculations involved (it's been years and I'm no longer involved with that area of operations), but you want to make sure your frequency matches the grid you're connecting with to within a certain amount. I think the stations which connect neighboring areas usually have extra monitoring equipment for a variety of energy accounting reasons, but also to measure the frequency on each side to see if it's a go or no go. I believe they use synchroscopes (https://en.wikipedia.org/wiki/Synchroscope) to do this.

Edit: it looks like y'all started talking about this below and I missed it, but the information is good, so I'm leaving it up.

    > Neighboring lines trip
    > As a last step you turn on your nuclear plants

Won't it make more sense to keep it running as usual maybe just using the power to generate steam that you throw into the atmosphere? ...mainly because stopping and restarting a nuclear power plant is expensive af.?

(It still boggles my mind why are electric grids so focused on efficiency at the cost of resiliency... with abundant nuclear + hydro you could easily over-produce a bit, I'm 99% sure that making electricity dirt cheap for consumers by gov controlled over-production would help fight climate change even if it appears wasteful: there will be huge incentive to replace petrol burning trucks, buses, cars etc. when electricity it dirt cheap... and then prices can be gradually ramped up after everyone made the conversion.)

The key factors seem to be:

1) the plant must be capable of running as an island. It seems not all of them are; I suspect the problem is the plant consuming a variable amount of self-power e.g. 1MW being connected to a 100MW generator results in a very unstable system.

2) whatever safety event causes the problem must not trip at the generators themselves but disconnect the plant cleanly from the grid. Since these events are by definition exceptional, this may be very hard to avoid.

> with abundant nuclear + hydro you could easily over-produce a bit

Both of these are exorbitantly expensive to build, and hydro requires specific geology.

How many people have a manual transmission car just so they can pop the clutch if the batter is dead.

A better question is, should we have nuclear power at all? I took classes in nuclear engineering, by friends are nuclear engineers, ect. but it's not obvious to me that cheap electricity is worth it.

Why would you say that? I mean sure, if you can't manage to get cost-effectiveness right then it's not worth it. But otherwise dirt-cheap and readily available electricity is absolutely needed to solve the world's problems! Think about what will happen when large swaths of land loose access to drinking water after climate + pollution wreck things up and the only alternative is ocean water filtration and desalination plants that will need huuuge amounts of power! Also think about massive irrigation and civil engineering projects that will be needed to fight desertifications etc. And to keep heavily populated costal cities above (the new, higher) sea level. That's terrawats upon terrawats of energy that will be needed to constantly shuffle earth and water around, and maybe even to power huge city-scale air filters etc.

We have and still are wrecking up the planet to such a high extent that sooner or later the bill will arrive and we'll need to put heavy effort into re-terraforming Earth to keep it inhabitable... And there really is no other answer than nuclear here, while cities filled with LED-lights and low powered devices and highly thermally efficient buildings can run on renewables, heavy civil engineering projects and land decontamination and producing drinking water etc. will need waaay more power than our blanket of communication equipment and drones.

We could decide not to, but then we'll end up fighting for resources and instead of nuclear power with it's inherent risks we'll get... nuclear wars!

> reactors require triple redundancy for their power supply, including the grid.

Thanks for posting. I learned something interesting today.

I can imagine the consequences of a total loss of power in nuclear reactors...

I'm curious about the statement about renewables though because it seems that with wind or solar the production is very irregular even in normal operation. Doesn't that mean there has to be some technology to balance individual generator differences which would make re-starting those installations easier?

Re: Nuclear, how much energy is required to start up a nuke plant? If decaying toxins are a risk shouldn't those be prioritized? I might misunderstand what you mean by toxins, does that reduce production for some time?

This proves problematic during reactor startup. Since it's absorbing neutrons, the controllers would need to retract the control rods further out than during steady state in order to achieve a replication rate of 1.0. However, xenon-135 stops being a nuclear poison after absorbing its neutrons... which means the replication rate will increase, and the reactor might run away.

This is part of what happened in Chernobyl, though they'd also disabled most of their safeties. (And dismantled the others.)

Regardless, nuclear poisons are something to keep a very careful eye on.

Xenon poisoning was discovered at the Hanford, Washington, B reactor during the WWII Manhatten Project.

https://www.osti.gov/opennet/manhattan-project-history/Event...

In future I would expect renewable and storage elements to provide black-start frequency reference.

There are also organizations like the RTO/ISO that also act like air traffic controllers for the grid by keeping it reliable. They also are usually responsible for committing and dispatching generation for dozens of utilities collectively to save money (by banding together, you need to carry less reserves or backup power and can leverage economies of scale). In North America these organizations are (CAISO, SPP, ERCOT, MISO, PJM, NYISO, and ISO-NE). All of these organizations practice black-start drills (I used to help run them) and have NERC certified operators.

Close. My dad's a power engineer, specifically grid protection.

The transmission level engineers and operators will call the shots as far as how the high level stuff goes down. In other words they'll issue a directive to get unit x online and then close in breaker b and clear a path to unit y while picking up load at z. The actual process from the end of the people doing that work is likely to be significantly more detailed.

And has a nosy son ;)

Each utility generally has its own black-start procedures. One utility might start two diesel generators (only really used for black-start) and then slowly start energizing the path to various loads and other generators. That utility and it's parent RTO/ISO train on that scenario several times a year and have the procedure printed in a binder within reach.

Something I've found interesting is that many of these plans are ~35 years old and we're created before computers were super common place for these kinds of things. The point being people weren't doing graph and optimization algorithms to determine the optimal paths. There has been a little university research and national lab research into this recently and I've heard they've helped certain utilities that they worked with determine some improvements, which is always exciting.

My dad's a power (line protection) engineer, everything I know is from talking to him.

I love how the people coming from different fields invented their own word for "the process of starting a system through external intervention because it cannot support itself at this point".

Kickstart. Jumpstart. Bootstrap. And now I learned a new one: blackstart.

[1] Synchronizing AC generators -- Part 1 (introduction and sync lamps)

[2] Synchronizing AC generators -- Part 2 (strobe light view)

[3] Synchronizing AC generators -- Part 3 (sync and unsync)

[4] Synchronizing AC generators -- Part 4 (phase deviation)

The first video explains all the concepts and shows a sync, but the later videos bring additional conditions and detail.

[1] https://www.youtube.com/watch?v=RGPCIypib5Q [2] https://www.youtube.com/watch?v=sFohkp2aaU4 [3] https://www.youtube.com/watch?v=GRk_qJxaxh8 [4] https://www.youtube.com/watch?v=RT1ySBc-Bls

Phase is monitored on both sides of the connection, and high voltage contactors complete the circuit when tolerances are met.

In case of phase mismatch, you have a short circuit and the weakest element along the short gets fried.

E.g., will an out of phase turbine have a physical resistance against it, as a consequence of the rest of the grid?

This kills the generator. Also, less amusingly, anyone nearby at the time.

If it's a larger mismatch, then that's more likely to fry the wiring before too much other damage happens. On rare occasions generators get connected while they are (exactly) 180 degrees out of phase; this is a dead short, and its internals will instantly vaporize. The vaporized copper may then proceed to explore the outer limits of the generator's physical integrity.

(Disclaimer, etc: I've only worked with DC generation, AC would be somewhat different especially if it's three-phase.)

Diesel units in hospitals will sync to each other as much as 30 degrees out of sync because just get thing online because it the load is critical.

I've also been told two second hand reports of the breaker closing drastically out of sync. One was a person closing the breaker manually when the unit was stopped. It essentially just puts a fault on the machine and it trips right away. Another case the close circuit was poorly designed and has only one relay, whose contacts eventually failed but happened to fail welded closed, and in conjunction with a poorly maintained breaker that developed a variable delay between the close coil being energized and actually closing. It made a big boom like faults do but the unit synced after the relay was replaced and breaker maintained and is fine.

Maybe medium and larger units with more inertia have tighter tolerances. Over certainly heard of broken shafts and generators being ejected through concrete walls.

A nice party trick is to do phase matching instead of syncing with slip so I can keep the unit exactly in phase with the grid as long as the grid doesn't wobble too much. This keeps the sync scope needle exactly at 12 o clock so you can close the breaker to put the unit online at will. Unfortunately i don't often get the budget for The additional inputs to get the nice phase difference signal so I can't perform my trick.

Well, when it was connected about 30 degrees out of phase, the entire unit pretty much exploded. It tore out of the concrete base, and was found to have rotated nearly halfway to the "correct" position.

I haven't seen anything nearly that dramatic myself, so I can't be sure how much it's exaggerated. There's probably a report somewhere.

And I love that party trick. There's no direct equivalence in DC transmission, of course, but I've done similar things with voltage-matching in Electrical Age. It always amazes me how much that game ends up reflecting reality.

Generally systems try to avoid too much loop power in the first place for this reason, setting up a directed but acyclic graph with some redundancy links carrying zero or near-zero power as standby.

I've been to about 30 stations under 50 MW and the biggest station load might have been 300 kW if they had all the heaters, fans, pumps, crane, welders, air compressors, and gates operating at the same time.

Large scale transformer blowouts are a nightmare scenario due to how long they take to replace / repair.

For fast transformer replacements we rely on two paths, either system spares (identical units, sometimes refurbished or like in kind replacements) and more recently mobile resiliency transformers. Resiliency transformers are trucked in on semis and can operate at 69, 138, or 345KV in our case. (There's some lightweight reading on them here: https://www.tdworld.com/overhead-transmission/siemens-introd...)

Years back I remember ConEd was replacing two or three very large transformers down the street of a large venu I was working an event. Turns out we worked the same event the next year and amazing the same day ConEd again had the second transformer scheduled to moved into place. What an operation. Huge trailer from Bay Crane along with one of their giant hydraulic cranes. Was told the transformer weighed in at 240,000 pounds.

I know a few people in the ConEd trenches, one guy is a lineman who does all the >=69kV stuff and the other guy worked in substations and was in mott haven last I spoke to him (cushy job).

I'm always amazed how much new tech con ed has installed and how much old stuff is still kicking around. I live in south queens and we're all 4kV overhead fed from area subs in turn fed by 27kV. Some of those 4kV poles have westinghouse transformers from the 30/40's still in service, glass insulators, and leftover wooden pins from older insulators. Some are so old the primaries have fallen off and just hang between two adjacent poles, some the crossarms are half falling off. Amazing this grid doesn't go down more. Then the next pole is brand new, composite strain insulators, cutouts, reclosers, metering, etc. All modern hardware. And I know some areas they upgraded the 4kV overhead to 27kV. The secondaries are in even worse condition, ancient cloth insulation half weatherd/baked off. But it all still works.

I really love how there's this massive network of cabling and systems which orchestrate the flow of power from multiple synchronized generators so it can get to your humble wall socket and charge a phone or turn a light bulb on.

(I'm not an EE, so not sure how many variables one needs to contend with in grid-scale transformer design)

Older stations were actually designed around the transformers in some cases and the physical configuration of a 50+ year old failed unit is rarely like those produced more recently. We might have a unit on hand that is electrically equivalent but impossible to install without essentially rebuilding the entire station.

To give a rough overview of the range of transformers on our system, the first thing we care about is the voltage and MVA rating, then whether the bushings are open air or gas insulated, what the winding configuration of the primary and secondary are (delta-wye being most common), whether the unit it is replacing has a tertiary winding (sometimes at transmission voltage, other times distribution or for light and power to the station), and then maybe what accessories it comes with such as current transformers for protective relaying.

A perfect like in kind replacement could possibly be done entirely by construction using proper procedure of documenting their disconnects if there were no structural modifications required to move the units in/out. This is rarely the case however and so we need to check everything from phase clearance on making the high voltage connections to whether disconnecting a failed transformer will trip out another perfectly healthy one, possibly making the situation much worse.

All that being said, we make every attempt to triage the work required and get information to construction as soon as possible so that no time is wasted waiting for engineering. We can have a conceptual layout circulating within engineering in a few hours and out to the field by the next day. Field technicians can tag and disconnect terminals right away while we figure out how to wire up the new unit. Perhaps in a perfect situation you might get all this done in a few days with all hands on deck but that's more luck than aptitude.

Interesting. I had always wondered how utilities interact with local permitting. I had assumed they would work something out to have standardized permits for common work which they could leverage during incidents but that seems that's not the case.

https://en.wikipedia.org/wiki/Solar_storm_of_1859

One of the biggest issues is that traditional resources (coal, nuclear, gas) can run for long periods of time and in many places have a pretty reliable fuel supply. Renewables are great, but you could be generating lots of power one minute and then it's all gone the next hour. This causes a lot of stress on traditional resources which need to be ready to counteract this. Coal plants and gas plants were never designed for that either and it's causing issues.

That is ignoring issues like Whitefish Energy among other things. Very different situation.

edit: yes, we know that nothing anywhere near comparable to hurricane Maria hit them.

There's been grid problems since I have memory. Doesn't surprise me that the whole thing eventually went down.

Maria wasnt the only hurricane that left us in the dark. George and Hortense were a 2 month ordeal with no power (aside from small gasoline generators) in the town I grew up.

Source: born and raised here