The issue with this specific outage is that it was badly managed.

The main gripe people and authorities have with what happened is that key infrastructures immediately lost power. When (only) 5% of demand is turned off this should not impact railways, airports, and hospitals.

Beyond the initial loss of power other problems amplified the consequences: For example, trains could not just restart but required specialist staff to attend each of them to get them through a specific procedure, which turned a short power loss into hours of delays.

This highlights the importance of looking at systems globally. The grid is just one part of the system.

A lot of the headline grabbing stuff wasn't the grid's fault though:

* the railways didn't have traction power disconnected -- but one type of (new) train saw the within-specification frequency drop, shut down, and in many cases needed a fitter to go out and restart it. Everything else behind a dead train was stuck. I imagine the train company are discussing this with their supplier...

* Ipswich hospital had a power outage, but that was because their own systems disconnected from the grid when they saw the frequency drop, and then they had a problem with one of their generators.

*One regional airport seems to have been accidentally not on the list of key sites not to disconnect -- sounds like an admin snafu, and has now been fixed.

None of these seem like "bad management" -- they are a collection of relatively minor, entirely unrelated, problems of the sort you're likely to run into if you have to activate an emergency procedure you haven't needed to use in a decade.

I read your list and see a bunch of 'technical debt.' Trains put into operation without the ability to survive a power transient without intervention. Life critical backup power system failures. The set of critical sites is not being maintained properly. In my mind these failures are are not entirely unrelated; reliable power is taken for granted and the testing and maintenance necessary to cope with power outages is neglected.

This is not technical debt. These are signs of oversights or potential poor management. Indeed that's what neglecting these procedures is.

This is why a comprehensive investigation is needed to ensure the details are known and all lessons learned or, unfortunately, re-learned.

Its shockly poor procurement - those new trains are not fit for purpose.

Do you mean that they don't meet the requirements of the tender spec ("not fit for purpose") or that the spec in the tender process was deficient ("shockingly poor procurement")?

Both the older electrostars where far better

> Trains put into operation without the ability to survive a power transient without intervention.

The trains were built to a environmental tolerance spec, and shutting down when that spec is exceeded seems like a reasonable thing to do. Having an expert test that all the safety systems are still working properly after they've been exposed to conditions they weren't designed for also seems prudent.

Do you think people should pay higher ticket prices / more tax to fund trains that can tolerate a wider range of input frequencies, given how rarely these sorts of events happen?

> The trains were built to a environmental tolerance spec and shutting down when that spec is exceeded seems like a reasonable thing to do.

Yes, they were built to a spec. According to the official technical report[1] these trains DID NOT experience anything outside that spec. The shut down was not supposed to occur. The trains are simply defective and both manufacturer and operator testing failed to reveal these defects. There is no other reasonable conclusion. Defective, inadequately tested trains.

> Having an expert test that all the safety systems are still working properly after they've been exposed to conditions they weren't designed for also seems prudent.

This bit of guesswork is also contradicted in the official report as the manufacturer has committed to providing a system update to permit operation to resume WITHOUT any of your supposed 'prudence.' Resuming operation on power recovery without intervention by technicians is the specified, expected behavior. This is a separate defect that also survived whatever obviously inadequate testing regime was supposedly in effect.

> Do you think people should pay higher ticket prices / more tax to fund trains that can tolerate a wider range of input frequencies, given how rarely these sorts of events happen?

Since the official conclusion and subsequent response by the manufacturer both show the trains were specified to tolerate these conditions the extraordinary costs you imagine are a fiction; the cost has already been paid. No higher ticket prices required. And since these conditions were specified we also know they were indeed anticipated despite their infrequency.

Both the manufacturer and the operator failed here. They got caught with their pants down. You may not like hearing that for whatever reason but that's the bottom line and inventing excuses won't change it.

[1] https://news.ycombinator.com/item?id=20973513

The GP says that the frequency drop was within-specification, not exceeding spec.

Which specification though? At the time they were being designed and procured, I think the specified tolerance for frequency change rate on the grid was 0.125Hz/s, but this was relaxed in 2014 to 1Hz/s (see https://www.nationalgrideso.com/document/10771/download). How much the effects of relaxing this on existing equipment was considered, it's not clear.

Similarly, the grid's license gives them a ±1% variation around 50Hz (so ±0.5Hz from https://www.nationalgrideso.com/balancing-services/frequency...).

So we have a power anomaly where the frequency moved >0.125Hz/s and dropped well below the 49.5Hz license threshold. I don't know what the procurement spec was, but I'd be very surprised if this event was "in-spec" for the trains, so in the absence if this it's probably reasonable to say that GP is incorrect.

Transformers get quite unhappily melty if you drop the frequency too much, so if your cooling system isn't designed to cope with this, it's probably better to shut down than have an exciting fire.

The grid's license only requires them to be within ±0.5Hz of the nominal 50Hz for at least 95% of the time. Anyone designing a system which needs to operate reliably ideally ought to design it to operate over the full 47.5-52Hz range, or at least not fail so completely it cannot be restarted. Low frequency demand disconnect only starts at 48.8 Hz, ao any system that can't handle that is effectively setting itself up to be the first thing to fail when there's a power shortfall. This is utterly inexcusable for a train given the amount of disruption caused by multiple lines being clogged by failed trains in multiple places.

I don't disagree that a train that can tolerate a wider range is desirable, but lots of things are desirable. It's also desirable to give everyone their own carriage, or have a free bar on every train. But at what cost?

* Wider tolerance means heavier electrical and cooling systems, increasing both capital outlay and operating cost for both track wear and maintenance.

* The trains procured were essentially "off the shelf" from Siemens and would have been designed and manufactured for other operators before the DfT put the tender together. A wider operating tolerance would have therefore either ruled out the Siemens trains, or at least significantly upped the cost as they would have had to redesign them.

An annual season ticket from St Albans to London is about £3,600. How much more should that passenger (as well as the taxpayer) be paying so that they can travel on trains that work when this type of event happens? How often does this type of event happen?

> This is utterly inexcusable for a train given the amount of disruption caused by multiple lines being clogged by failed trains in multiple places.

People being stuck on trains for a few hours is pretty far down the severity list of "things that could go wrong" when the power goes out. That said, what I do think the outcome will be here is they'll figure out if/how the driver can self-reset this type of issue rather than requiring a fitter to travel out.

In a way these events become a blessing in disguise because of those things. We've had to deal with similar crisis situations in our supply chain. It really helps to test your procedures and operations fixing stuff along the way. The team also gains experience and becomes much more prepared for worse events should they come.

I would go as far as suggesting that these things should be routinely tested. In the UK most public buildings test fire alarms once a week; a programmed plan of localised, managed blackouts, to be carried out once every few years, would probably help in the long run - particularly as the grid is rapidly changing as described at the end of the post.

Of course they should be routinely tested. This is not a new idea and is an established best practice.

This is why I am very surprised when I read that hospitals generators failing to take over is just "one of those things".

It's like the 'old' saying that you don't have backups until you test recovery from backups...

We didn't hear about all the other hospitals that easily switched over to generators, just the one that failed. There's room for investigation there, but it's on the site, not the system.

Definitely. We use seasonal demand peaks to learn how to cope with extremes. But if we didn't have those plus strikes I'd definitely be planning a few test runs.

I did mention the railways and indeed the importance of making sure the whole system can cope, which means what you list in addition to the grid in isolation.

> None of these seem like "bad management"

They do seem like bad management because to me it looks like these issues were not taken seriously.

> problems of the sort you're likely to run into if you have to activate an emergency procedure you haven't needed to use in a decade.

That's bad management right there if an organisation's emergency procedure has been collecting dust for a decade.

> They do seem like bad management.

But not bad management by the Grid.

Including by the grid because key infrastructures did lose power.

But in general a worrying aspect of what happened is that there seems to have been oversights all over the place (grid, hospitals, railways, etc).

How many 9s do you want from the grid?

There are lots of reasons why hospitals lose power, which is why they all have backup generators.

Ipswich's backup generator had a faulty battery. This is nothing to do with the grid. This is entirely the fault of the hospital trust. Most of the hospital's backups worked.

https://www.ipswichstar.co.uk/news/ipswich-hospital-blackout...

> A spokeswoman said: "Our initial investigation has shown that at 4.52pm on Friday, August 9, we lost our electricity supply until 5pm. We have 11 generators at Ipswich Hospital and all of them worked immediately.

> "What we have found is that the battery which automatically operates one of the switch over systems from mains power to generator power was faulty despite being within its recommended life.

Here's the trust document talking about the new installation when it was put in.

http://www.ipswichhospital.nhs.uk/aboutourhospital/Documents...

> Electrical High Voltage Infrastructure was a critically important project to the Trust in providing a resilient high voltage electrical network to the hospital. Through a range of electrical projects, including the replacement and upgrading of outpatients generator ‘C’, the maternity generator project and the upgrading of the high voltage switchgear for the forthcoming biofuels project, this represented a total overall investment in excess of £2m. Virtually all the Trust’s electrical switchgear has now been improved and modernised. Along with the new backup generators the hospital is protected for years to come and a major significant risk to hospital has been dealt with.

It's clear that there were issues at hospitals (as expected cynical me thinks, knowing British healthcare...) but the fact that key infrastructure lost power does points to issues at the grid level as well (because sites that lost power did so based on grid procedures).

When they cut the power to an area, they can't selectively keep isolated locations active unless they are isolated in that they operate upon their own spur. That is why Hospitals would be deemed critical and important not to cut out, they are handled by onsight failover power supply in the form of backup generators.

In this case, it was like asking somebody to be able to catch a ball, they say they are ready and yet fail to catch that ball when you thru it to them. Is that the fault of the person who throws the ball or the person that was not prepared to catch the ball when they said they could.

In this instance it falls into the infortunate situation, and from what I can tell, you can't lay blame onto anybody or party for the failure and it is just one of those things. Sure leasons will be learned, but from the grid procedure in this hospital case, there will be no change I can see that would be learned. However the Hospital will be learning from this and may have more frequent testing and checks of their failover, may make changes. So if blame is to be leveraged upon anybody - you could say it would be from the party that had to make changes. But in this instance, would that be fair. Hence I would call this unfortunate and whilst lessons will be learned, you can not lay blame out of those lessons all the time. However - if it happens again and those same mistakes transpire - then, maybe you have some grounds to lay some blame.

The actual report says that Ipswich hospital wasn't part of the load shedding; so it's not clear why the generators had to cut in in the first place yet.

When the frequency is nosediving and below some threshold it is a good sign that you want to switch to your back up power immediately. Even if there isn’t ultimately a cut, it is bad quality power and a sign of system instability

I'm sure that they will learn lessons that railways, hospitals, airports should perhaps not be the first to lose power, that trains should be able to restart quickly, and that generators should be comprehensively tested to respond to real situations, and it will be no-one's fault.

I fully agree with you that this is the likely outcome.

Reading this exchange as a third party, both sides make fair points, but one side is being excessively downvoted, which is either unfortunate or possibly a regrettable sign of emotional over intellectual engagement in the discussion somewhere.

It is rather odd, indeed, but usual. Their often seems to be an "incorrect" opinion.

It seems that one of the commenters work for the National Grid or related so might be a bit partial...

Sounds like a good chance to test the systems

Ah I wondered why the ST Pancras line was in such a mess hours later

You're misunderstanding how it went. The 5% that's mentioned as the Low Frequency Demand Disconnection scheme, who were disconnected in managing the incident, is a scheme where very high volume users (those who have a meter reading taken every hour) voluntarily opt to have interruptible supply for better prices. When the grid needs to, they're given a power cut that they signed up for. Commonly used by big manufacturers, smelters etc. It's in their contract, expected and gets them lower electricity bills.

It helps prevent the grid tipping into catastrophic shutdown when there's no backup generation left, or something major has tripped, as part of demand management. If it hadn't happened the consequences would have far more headline-worthy.

Railways, hospitals and the like that encountered issues were part of "the rest". That is not part of the 5% turned off, but the small part of the 95% "rest" that can't cope with some (larger than usual) drop of frequency. If they're buying trains that can't cope with a frequency slump or backup systems that aren't tested that's outwith grid control. That's a sign their systems need to be improved, and tested properly too.

Here in Germany large industrial customers often get a cheaper electicity price in return for being the first to be cut in a load shedding scenario. If you are something like a steel mill then cheaper electicity quickly pays for slightly more frequent blackouts; and everyone else gets a more stable electricity network.

That's the case in the UK too.

Most situations/responses to an issue in hindsight can always be managed better, being able to have a cookie-cutter solution that fits most cases is better than none. THough I'm sure such plans will be refined - leasons learned.

Thing is, a solution today might not be best for tomorrow as the grid's usage is not static, new this and that are added so the impact changes.

It is interesting that you mention the inability for trains to restart. That was educational and was due to some new trains and showed that the drivers did not have the skills to in effect - reboot the trains from a cold start. So less a grid fault and more a case of that it never happened before so the result was less prepared. Interestingly the previous generation of trains did not have this issue, which may explain why it got overlooked as something that needed attention until such an event like this.

As for hospitals, they all have backup generators, though if that generator fails to kick in as was the case in one I read about, that can and will cause issues. Again, may be a case of never needing it before and the combination of that and regular checks/tests presented the problem.

Remember - nothing is perfect, and many lessons (as with any outage in all walks of life) will be learned from this as with all previous outages. But equally have to remember that whilst you can have the same combination of events happen in say 5 years time and have a response that works for that combination learned from the last time. Things change, like that new housing estate attached to the grid, small nuances like that could change everything or nothing. You just don't know. Heck, could be newer issues, suddenly find that you have thousands of electric cars being charged at the time you cut them off now feeding back small amounts of electricity back into the grid as they never expected to loose power without a physical disconnect.

All the issues I mentioned are obvious, and so are the potential problems you describe. This is not a question of "it's easy to say in hindsight" as these are the key issues that must be covered by standard planning.

This blackout made a lot of noise because of this (apart from the actual disruption, obviously).

Do you have background in the field of giant power supply fault tolerance backup mishap management strategies, or something? Or are you just stating and assuming all of this is obvious, covered by standard planning, instead of easy to say in hindsight? Cause you're making a lot of statements, saying it is such, without backing it up. Maybe it is super obvious, but personally I'm impressed by the strategies that were in place.

Would you not prioritise an airport, a hospital, the railway, over residential areas?

Do you not think that trains should be able to resume ASAP? Do you not think that "it never happened before" is not a valid reason to be unprepared?

Do you not think that "we never had to use it before" is not a valid reason to explain why an emergency generator might not work when needed?

What backing up do you need?

"Would you not prioritise an airport, a hospital, the railway, over residential areas?" Yes and they do.

As mentioned, things get added to the grid all the time, case of the airport was that it was not on a priority list. Yes that is an oversight, but who's fault and accountability we do not know. The onus would of been in the airport to inform the power distribution and if that failed to happen then the airport would be the ones with the hands up, if they was and the distribution failed to add it, then their hand would go up.

But hospitals as I mentioned operate and a designed and planned for such issues with them having backup power supplies. Which is the solution for many installations as many will not have or justify the cost of being upon their own distribution spur.

As for the "we never had to use it before" that's down the testing and covered in previous comments fairly well without repeating it again.

But if you wish to focus upon being able to point a finger at somebody and say it is there fault, then fine. But I and I'm sure many others would not be focusing upon that, though I can appreciate that mentality due to the way media/tabloids cover such and all issues in life. After all, power cuts today are not that common on this scale, decades past and in less developed countries they would be so common that they wouldn't even make the news.

But remember, if the weather forecaster says it will not rain and you go out and a light shower makes you wet - would the forecast be wrong or would we have the mentality to go, that mistake was within acceptable parameters and just one of those things. It gets down to what we are used to and what we accept. So yes, I get that blame need to be lain in many eye's but is that fair all the time?

But then - nothing is perfect - better to focus on getting closer to perfection than expending energy stagnating as too busy pointing the finger.

I'm pretty sure there is a lot of people who is interested to know if this is someone's fault, and I don't think it has anything to do with how media covers this issues. I think it's a good thing, in fact.

This failure may be within acceptable parameters and it may be simply meaningless to try to find someone truly responsible for it. But you seem to be unreasonably reluctant to believe that maybe it's not.

I understand that it's interesting to analyze what happened from a more technical perspective, but of course a proper investigation has to be conducted to find the responsible if there happens to be one (or more).

"But you seem to be unreasonably reluctant to believe that maybe it's not."

WE have covered many area's, trains, planes and hospitals (maybe a movie there). Sure I'd rather focus on the technical aspects and not presume this was something totally avoidable. Rather look at each nuance instead of the binary blame game. Which is not clear cut. After all - we don't know what SLA's supply contracts are in place now do we. But nothing is ever 100%. But it is possible to argue both sides of the coin and without ALL the details, go around in circles.

If we really want to blame anybody - why not just blame God for allowing it to happen and move on. Leave it to the report, which will happen into the incident and associated fallout to cover all that and discuss any blame then, once all the facts are in. But remember - we do not know what the SLA levels are, so all those fallouts take on a whole new perspective of blame. Equally, you can have a problem and nobody is to blame - though God works for those, at least "Act of God" is actually used in insurance contracts (yes worked in that field as well disclaimer, though late 80's early 90's). https://en.wikipedia.org/wiki/Act_of_God

Please don't cross into personal attack here.

https://news.ycombinator.com/newsguidelines.html

I do not work for the grid and last involvment in that area was working for Eastern Electricity Board as an analyst/programmer upon a COBOL DPS7 mainframe circa first half of the 80's and worked upon a new plant management system mostly (distribution level so upto attaching to the main grid). I've had no involvement in that industry since then. I have also worked for the Department Of Health, RAND, and numerous other industries and involvement.

I have no axe to grind. Hope that adds some clarity.

Several people in this thread have said that you are wrong to place blame entirely on the grid. It's unlikely that they all work for power distribution companies.

I have never placed blame entirely on the grid... Can we not lower ourselves into this sort of sordid methods of trying to put words into others' mouths?

You protest a bit too much at this point, methinks.

I am commenting that the scale of the disruption resulting from (only) a 5% drop does not seem reasonable and that some of the issues should have been better handled or avoided altogether.

> But I and I'm sure many others would not be focusing upon that, though I can appreciate that mentality due to the way media/tabloids cover such and all issues in life

That's a personal attack quite out of order.

> > But I and I'm sure many others would not be focusing upon that, though I can appreciate that mentality due to the way media/tabloids cover such and all issues in life

> That's a personal attack quite out of order.

Not really. It's just a societal commentary on how we've conditioned ourselves by the media we consume into passing quick blame instead of taking a slow, reasoned approach to analyzing the event, seeing the compounding factors at stake and only in the face of gross negligence assigning blame.

It is a personal attack and it is not what I have written so also misrepresent my comments.

Neither of which are acceptable.

> seeing the compounding factors at stake and only in the face of gross negligence assigning blame.

You are entitled to have your views. Claiming that there is no blame to assign until it reaches criminal liability is rather odd, though.

I am NOT a safety engineer, and I'm aware how complex that field is, so no, I don't believe that my personal hunches about what might have been a good idea intuitively are somehow better than these people whose actual job it is.

Hence my question about your background. But you implicitly answered that, too.

A potential argument against transit reliance, or at least trains relying on the grid to keep running (instead of using batteries or diesel). Personal transport options (cars, bikes, horses) are a decentralization success story when the grid goes down.

It may well be that cars with large batteries enable a smarter grid that can store surplus generation at off-peak and buffer the peaks better. After all - not everybody drives their car at the same time, and some access to a cheaper rate of electric for allowing your car to be used in such a way when it is plugged in, is certainly one avenue that may well open up more, as we see more and more electric cars.

Though the swappable battery charger offerings do seem to be more suited in accommodating such avenues and removes the worry of battery lifetime cycles from the owners. Worry that their car acts as a grid battery for a month when on holiday and reduces it's life beyond the cost of saving. So will be a case of getting the balance right. Hence I feel the swappable battery offerings may be at an advantage here and opens up making such solutions more cost effective to run.

It also highlights one of the most important characteristics of energy grids: frequency sensitivity.