News for hackers. Stuff that matters.
Re:Analysis: Texas power outage leaves officials working pol... by adolph on Wednesday February 17, 2021 @07:27PM (#26173820)
Better explainer:

https://www.texastribune.org/2021/02/17/texas-power-outage-a...

It appears that wind and solar power produced more electricity this week than they were expected to produce, but at this time of year, that’s a relatively small part of the expected statewide generation. Generation from gas and coal plants was far below expectations, causing supply to fall as demand rose. How much of that is bad design — a case where the system should be reworked — and how much was operator error is still to be sorted out.

As someone in the Houston blackout and hearing a bit of the initial blame-game: I’d like to find out what failure modes were gamed out, what likelihood’s were assigned to each and how and what mitigation was chosen with the knowledge that between time, money and people not all failure scenarios can be covered.

For example: water vapor in methane (natural gas) freezes and causes sensor and valve problems which have shut plants down. That seems predictable to a point, something that operators with experience in other climates would know about. On the other hand, the mitigation strategy for that failure may run counter to fixes for hot weather problems.

Re:Analysis: Texas power outage leaves officials working pol... by SiempreViernes on Wednesday February 17, 2021 @07:48PM (#26173995)
Well, apparently this was all reviewed in 2011, after a million Texas homes lost power due to a cold snap, so indeed most of the problems today were not only foreseen long ago, they were experienced.

Since then I guess nobody bothered to do anything? Supposedly it's all described in this here document (I can't actually open it, probably because someone in ERCOT is real scared of GDPR): http://www.ercot.com/content/meetings/other/keydocs/2011/201...

Here's a twitter thread by some rando who remembered what happened in 2011 and dug out the report: https://twitter.com/kidstatic/status/1361914818307981314?s=2...

Re:Analysis: Texas power outage leaves officials working pol... by throw0101a on Wednesday February 17, 2021 @08:15PM (#26174220)
> Since then I guess nobody bothered to do anything?

The folks in the El Paso area (which is not regulated/managed by ERCOT) are going just fine it appears:

> El Paso Electric said they always try to prepare for the future and after a winter storm in 2011, the utility company worked towards replacing and upgrading their equipment. Many generators now have antifreeze protection.

* https://kvia.com/news/el-paso/2021/02/15/el-pasos-not-seeing...

Re:Analysis: Texas power outage leaves officials working pol... by adolph on Wednesday February 17, 2021 @09:45PM (#26174853)
Here is an interesting thread by an academic:

https://twitter.com/JesseJenkins/status/1361691683222654980

Re:Analysis: Texas power outage leaves officials working pol... by fny on Wednesday February 17, 2021 @07:54PM (#26174044)
The imprudence is chilling. From that doc:

"THE EVENT

The strong arctic front that arrived in North Texas on February 1, 2011 was the most intense cold wave for the majority of the state of Texas since December 1989. The event was so severe that the National Weather Service issued a rare Wind Chill Advisory for much of the state. The arctic front reached deep into Texas and sustained below freezing temperatures as far south as Corpus Christi. Much of North Texas experienced record setting sleet and snow, producing blizzard conditions and temperatures well below freezing for up to 100 hours. Snowfall occurred as far south as San Antonio, with snowfall totaling up to seven inches in North Texas. This winter storm was also accompanied by sustained winds of 30-40 mph with gusts as high as 51 mph.

The icy conditions of this winter storm resulted in the closing of both DFW Airport and Dallas Love Field. In addition, Dallas-Fort Worth area water utilities reported hundreds of water main bursts during the event, compared to expected four to five on an average winter day. ERCOT also reported a new record winter peak of 56,334 megawatts.

The winter weather event of February 1-5, 2011 was determined to be a one in ten year event for some regions of Texas in terms of low temperature extremes and duration. Lows during the period were in the teens for five consecutive mornings and there were many consecutive hours below freezing temperatures throughout Texas. Taking these temperature extremes into account, and coupling them with the sustained winds of this event, it is estimated that the resultant convective heat loss (wind plus ambient temperature) event suffered by many Texas generation facilities approached a one in 25 year severity. (Reference: Weatherbank, Inc.)

Generating unit issues that affected reliability were mostly attributed to frozen instrumentation due to convective heat losses greater than instrumentation design, faulty instrumentation heat tracing or compromised insulation. It is normal practice for the operating units to add incremental protection against these extreme winter weather events. Typical examples are wind breaks and heat sources applied to high risk areas.

In addition, the cold and wind affected the units that were not operating. Units in cold start status are less prone to derates/trips if they are dispatched to minimum stable load (MSL) before the onset of extreme winter weather. Bringing all units to MSL gives ERCOT maximum flexibility in responding to grid emergencies.

HOW IT AFFECTED ERCOT

The high load demand, coupled with generation issues within ERCOT, resulted in actions from ERCOT that triggered an Operating Condition Notice (OCN) issued for cold weather at 08:55 on February 1. The event level escalated to an EEA2A (Energy Emergency Alert) declaration at 05:18 on February 2 to deploy responsive reserves, followed quickly by an EEA3 declaration at 05:58 on February 2 to begin shedding firm load to maintain system frequency at 59.8 Hz or greater. The event continued until ERCOT cancelled the EEA at 09:58 on February 3. "

Re:Analysis: Texas power outage leaves officials working pol... by worker767424 on Wednesday February 17, 2021 @08:09PM (#26174167)
> determined to be a one in ten year event

Good estimate.

Re:Analysis: Texas power outage leaves officials working pol... by adolph on Wednesday February 17, 2021 @09:34PM (#26174773)
This is great stuff, thanks!
Re:Analysis: Texas power outage leaves officials working pol... by Pfhreak on Wednesday February 17, 2021 @07:32PM (#26173865)
I suspect that it wasn't the mitigation was too challenging, just costly. Someone looked at the map and said, "Texas? You'll never need to be handling subzero temperatures over a long time period."

I don't even necessarily believe it was malicious (unless you consider cost cutting to make a profit malicious, which, maybe you'd have an argument there), just potentially lazy.

This is the sort of thing we should have some baseline regulations to support -- critical infrastructure should be designed to support a temperature range of X to Y, where X is a record cold times some factor and Y is a record heat times some factor.

Re:Analysis: Texas power outage leaves officials working pol... by throw0101a on Wednesday February 17, 2021 @08:10PM (#26174181)
> This is the sort of thing we should have some baseline regulations to support -- critical infrastructure should be designed to support a temperature range of X to Y, where X is a record cold times some factor and Y is a record heat times some factor.

Meanwhile the folks in the El Paso area (which is not regulated/managed by ERCOT) are going just fine:

> El Paso Electric said they always try to prepare for the future and after a winter storm in 2011, the utility company worked towards replacing and upgrading their equipment. Many generators now have antifreeze protection.

* https://kvia.com/news/el-paso/2021/02/15/el-pasos-not-seeing...

The first step is making sure you have regulators who are somewhat clued in.

Of course being hooked up to the national grid could be useful as well, so as to allow for energy inputs when your own infrastructure is having problems. Though that would mean dealing with the Feds, which seems to be undesirable:

> Former Texas Gov. Rick Perry says that Texans find massive power outages preferable to having more federal government interference in the state's energy grid.

* https://cbsaustin.com/news/local/rick-perry-says-texans-woul...

Re:Analysis: Texas power outage leaves officials working pol... by adolph on Wednesday February 17, 2021 @09:20PM (#26174679)
That’s great that El Paso got this failure mode right. Are there other modes that they de-prioritized?

El Paso Energy is interesting bc it represents the old school community monopoly. I wonder if the JPM buyout will help or hinder its mission.

https://en.wikipedia.org/wiki/El_Paso_Electric

Re:Analysis: Texas power outage leaves officials working pol... by chooseaname on Wednesday February 17, 2021 @09:03PM (#26174579)
> Former Texas Gov. Rick Perry says that Texans find massive power outages preferable to having more federal government interference in the state's energy grid. * https://cbsaustin.com/news/local/rick-perry-says-texans-woul...

I bet they don’t. If they did they don’t now.

Re:Analysis: Texas power outage leaves officials working pol... by JetSpiegel on Saturday February 20, 2021 @09:11AM (#26204185)
Rick "What is the Department of Energy? while being head of the DoE" Perry.
Re:Analysis: Texas power outage leaves officials working pol... by shawnz on Wednesday February 17, 2021 @07:44PM (#26173965)
> I don't even necessarily believe it was malicious (unless you consider cost cutting to make a profit malicious, which, maybe you'd have an argument there), just potentially lazy.

Playing devil's advocate here: It might not have been malicious or lazy, but actually just good judgement in avoiding unnecessary waste and getting those services deployed sooner for less cost to the customer.

Of course it looks bad now that the situation they didn't expect to happen has happened, but in the long term, were the losses worse than what they saved by not implementing extreme cold weather mitigations?

Re:Analysis: Texas power outage leaves officials working pol... by bcrosby95 on Wednesday February 17, 2021 @07:54PM (#26174039)
This might be a reasonable excuse if they didn't have the same problem in 2011 and weren't warned it could have been much worse.

But they did. And were warned. And they ignored it.

Re:Analysis: Texas power outage leaves officials working pol... by shawnz on Wednesday February 17, 2021 @08:12PM (#26174202)
However many times it happened, that still doesn't tell us whether the costs exceeded the savings. "Twice" isn't much more often than "once"
Re:Analysis: Texas power outage leaves officials working pol... by Twixes on Wednesday February 17, 2021 @08:50PM (#26174485)
Well, of course costs of resiliency would practically exceed savings for energy companies. But the costs of outages are tremendous for Texas overall. That's why critical infrastructure must be well regulated. Corrupt politicians are sure to protect profits of the energy industry, but in the end we all pay for the shenanigans that deregulation allows them to get away with.
Re:Analysis: Texas power outage leaves officials working pol... by shawnz on Wednesday February 17, 2021 @08:56PM (#26174535)
Lets say it led to no increased profits and the savings were entirely passed on to the customer (perhaps through careful regulation like you mention). Then wouldn't the costs be better for Texas overall?
Re:Analysis: Texas power outage leaves officials working pol... by Retric on Wednesday February 17, 2021 @10:11PM (#26175100)
No, the cost to produce electricity is largely independent of the value people receive from it.

Differences in incentives are why hospitals for example have backup generators. They might be paying 100x as much to get power from a generator for a few hours a year, but losing power would cost them vastly more.

That said, individual consumers are going to have their own personal preferences. For some people saving 10c a month on their monthly electric bill might be worth a once in a decade power outage. Some might even agree to save 1c/month.

Re:Analysis: Texas power outage leaves officials working pol... by Scalestein on Wednesday February 17, 2021 @09:13PM (#26174641)
I think there is another factor of Choice that needs to be considered here. If people were given the choice between losing power when it gets cold and saving money or paying more for more reliable power then it might be reasonable.

Power companies making that choice for their customers isn't fair. As a utility it should be expected that it is well run and stable. It isn't an easy market to have competition in due to the infrastructure meaning we can't have "cheap and unstable" options next to "expensive and stable". Where the state decides the utilities fall on that axis is the question here, I personally think they should be on the more conservative side and provide stability.

The other thing to consider are the huge knock on effects from losing power. I may be ok with getting extra cash in exchange for power outages but when the stop lights are out and restaurant freezers start thawing things get a lot worse.

Re:Analysis: Texas power outage leaves officials working pol... by shawnz on Wednesday February 17, 2021 @10:00PM (#26174993)
Good points. Although, it is of course impossible to have perfect service. Is a major failure once a decade not "well run and stable"? What level of service would be acceptable?
Re:Analysis: Texas power outage leaves officials working pol... by usefulcat on Wednesday February 17, 2021 @08:07PM (#26174144)
“but in the long term, were the losses worse than what they saved by not implementing extreme cold weather mitigations?”

A lot of people (and businesses too) have been without any power for several days. In the case of Austin, “a lot” appears to be around 40%. Many more have had power only intermittently. That’s got to be a pretty sizable hit to economic output. Not to mention all the busted pipes.

[edit: formatting]

Re:Analysis: Texas power outage leaves officials working pol... by shawnz on Wednesday February 17, 2021 @08:14PM (#26174217)
But is it greater than the impact on economic output of having done a safer, but slower and more costly expansion of the electrical grid? That's much harder to measure
Re:Analysis: Texas power outage leaves officials working pol... by pixl97 on Wednesday February 17, 2021 @10:21PM (#26175201)
Heh, this is a 100 billion+ disaster when it is over. There are so many things people have not talked about yet. Multiple blood banks have lost their entire cooling and flooded at the same time. A medical testing and samples lab has had similar issues. You've not seen that in the news yet.

The there are the things we do know, like the Samsung IC shutdown that will cost millions in itself.

Re:Analysis: Texas power outage leaves officials working pol... by usefulcat on Saturday February 20, 2021 @02:07PM (#26206855)
"By Friday, when Ercot declared that the emergency had ended, 14.4 million still lacked reliable access to public water supplies, and the crisis had already cost the state $50 billion in damages, according to Accuweather. The cost of power on Monday alone was $10 billion, according to estimates from Wood Mackenzie."

https://www.bloomberg.com/news/features/2021-02-20/texas-bla...

Re:Analysis: Texas power outage leaves officials working pol... by shawnz on Saturday February 20, 2021 @02:14PM (#26206911)
That is indeed compelling for the mismanagement argument.
Re:Analysis: Texas power outage leaves officials working pol... by boardwaalk on Wednesday February 17, 2021 @09:03PM (#26174584)
It seems like you're thinking purely of net economic impact.

As someone in Texas who just got their power back and no longer has running water, I'm having a real hard time not posting a really vicious retort here.

Pure capitalism without any humanism is really quite gross and frustrating.

Re:Analysis: Texas power outage leaves officials working pol... by shawnz on Wednesday February 17, 2021 @09:13PM (#26174644)
No, I am really thinking of net impact, economic and otherwise (but the parent mentioned economic output, so I was replying to that point)
Re:Analysis: Texas power outage leaves officials working pol... by Pfhreak on Wednesday February 17, 2021 @08:43PM (#26174436)
Were the losses worse? Yes, workout a doubt? People died. Pets died. Homes and water mains were destroyed. People have to boil their water. Having power wouldn't have prevented all that but it definitely would have prevented some.
Re:Analysis: Texas power outage leaves officials working pol... by shawnz on Wednesday February 17, 2021 @09:00PM (#26174563)
It's possible more people could have died from, for example, medical delays due to smaller hospitals from increased electricity costs over a period of decades due to unnecessary risk aversion
Re:Analysis: Texas power outage leaves officials working pol... by vkou on Wednesday February 17, 2021 @09:06PM (#26174600)
... What?

The cost of electricity is not a constraint on the size of hospitals. And there are plenty of states who are part of the national grid, with cheaper electricity than Texas.

Re:Analysis: Texas power outage leaves officials working pol... by pwinnski on Wednesday February 17, 2021 @09:22PM (#26174687)
It depends on how you price a human life. Give me that number, and I can start to answer.

More than 300 people have ended up in ERs in Houston alone for carbon monoxide inhalation as a result of misguided efforts to avoid freezing to death, I'm not entire sure how to allocate that cost, either.

1989, 2011, 2021. At some point, having been told twice now, and soon to be a third time, you spend the money to avoid the pitchforks.

Re:Analysis: Texas power outage leaves officials working pol... by minikites on Wednesday February 17, 2021 @08:34PM (#26174360)
>losses worse than what they saved

I'm sure "we saved a few bucks" will be comforting to the families of people who froze to death or died of carbon monoxide poisoning from this entirely avoidable disaster.

Re:Analysis: Texas power outage leaves officials working pol... by twalla on Wednesday February 17, 2021 @07:54PM (#26174040)
> This is the sort of thing we should have some baseline regulations to support

Texas runs its own power grid specifically to avoid doing that

Re:Analysis: Texas power outage leaves officials working pol... by adolph on Wednesday February 17, 2021 @09:55PM (#26174929)
Blackouts happen elsewhere too. New Orleans today:

https://www.wwltv.com/article/news/local/orleans/rotating-po...

at the direction of MISO, the Midcontinent Independent System Operator.

The company describes itself as similar to an air-traffic controller, monitoring the flow of electricity across 15 states and one Canadian province.

Re:Analysis: Texas power outage leaves officials working pol... by Spooky23 on Wednesday February 17, 2021 @08:00PM (#26174101)
The gaming was more about how to minimize capital expense to throw a few more pennies on the dividend.

This stuff isn’t rocket science.

Re:Analysis: Texas power outage leaves officials working pol... by adolph on Wednesday February 17, 2021 @09:14PM (#26174649)
Without verified data we are all speculating. Profit incentive is always strong no doubt.
Re:Analysis: Texas power outage leaves officials working pol... by dang on Wednesday February 17, 2021 @08:26PM (#26174298)
Ok, we've changed the URL to that from https://www.politifact.com/article/2021/feb/16/natural-gas-n..., which isn't working. Thanks!
Re:Analysis: Texas power outage leaves officials working pol... by K2h on Wednesday February 17, 2021 @07:53PM (#26174035)
I work with product that detects water vapor in NG - if you have an example where this contributed to this blackout please send!
Re:Analysis: Texas power outage leaves officials working pol... by mike_d on Wednesday February 17, 2021 @08:08PM (#26174153)
Methane melts at -295.6 F (It's cold in Texas, but not that cold!) so if NG facilities are having freeze ups it is absolutely due to the presence of water or hydrates.

Because people have died, and the energy industry is heavily to blame, you're extremely unlikely to get any information to confirm or even satisfy your curiosity until a federal investigation has completed.

Re:Analysis: Texas power outage leaves officials working pol... by jkhdigital on Wednesday February 17, 2021 @08:16PM (#26174236)
Or, you know, maybe nature played a role in those deaths too. It’s not like there’s just electricity lying around on the ground, free for anyone, and the evil energy companies are hoarding it all. Might be more helpful to temper the righteous indignation for now.
Re:Analysis: Texas power outage leaves officials working pol... by smaddox on Wednesday February 17, 2021 @08:37PM (#26174386)
Perhaps... Perhaps righteous indignation is not helpful. Perhaps some humility and resolve would be more helpful. Perhaps each and every Texan, myself included, should take personal responsibility for this failure to prepare for the consequences of this foreseeable (and [foreseen](https://www.houstonchronicle.com/business/energy/article/Tex...)) disaster, and resolve to prepare both individually and by electing competent governors who won't prioritize corporate profit over resilience.

> A hurricane is an act of God. But failure to prepare, when the necessity for preparation is well known—that’s sin. That’s failure to hit the mark. And the wages of sin is death (Romans 6:23). The ancient Jews always blamed themselves when things fell apart. They acted as if God’s goodness—the goodness of reality—was axiomatic, and took responsibility for their own failure. That’s insanely responsible. But the alternative is to judge reality as insufficient, to criticize Being itself, and to sink into resentment and the desire for revenge. - 12 Rules for Life by Jordan Peterson

Re:Analysis: Texas power outage leaves officials working pol... by adolph on Wednesday February 17, 2021 @09:33PM (#26174769)
As far as personal prep, things aren’t that different from a hurricane/flood event. Intermittent power is better than 2 weeks without like Ike. The cold is a new twist but not unmanageable. Judge Hidalgo seems up to the task and a few steps ahead of the mayor.
Re:Analysis: Texas power outage leaves officials working pol... by adolph on Wednesday February 17, 2021 @09:13PM (#26174636)
I heard it from the CenterPoint person on this podcast iirc:

https://www.houstonpublicmedia.org/articles/shows/town-squar...

Re:Analysis: Texas power outage leaves officials working pol... by pixl97 on Wednesday February 17, 2021 @10:27PM (#26175247)
Wells freeze in all the time. Around a decade ago I was working with an in law that was dealing with a frozen dewatering unit on a very large compressor. We also measure the water tanks get pumped back into the earth. High output wells need their water emptied every day or two. Guess what an ice storm prevents. On, and the water tanks will freeze and cant be emptied.
Re:Analysis: Texas power outage leaves officials working pol... by WillPostForFood on Wednesday February 17, 2021 @08:26PM (#26174306)
There are two ways to look at the Wind numbers. Texas was expecting 6 gigawatts from wind, and only got 1.5-2.5 gigawatts. So wind generation dropped 60-75%. That's a much bigger failure in percentage terms than from gas powered production. Gas was a majority of energy production, so it was much bigger in absolute numbers, but by no means did wind perform well.
Re:Analysis: Texas power outage leaves officials working pol... by bryanlarsen on Wednesday February 17, 2021 @08:44PM (#26174442)
According to the Ars article, you have that backwards -- 6GW were produced while only 2GW were expected.
Re:Analysis: Texas power outage leaves officials working pol... by WillPostForFood on Thursday February 18, 2021 @12:22AM (#26176219)
The Ars article (can you post a link to it?) is wrong if it says that. Wind generation collapsed as it froze.

Only 7% of ERCOT’s forecasted winter capacity, or 6 gigawatts, was expected to come from various wind power sources across the state.

https://www.texastribune.org/2021/02/16/natural-gas-power-st...

Wind shutdowns accounted for 3.6 to 4.5 gigawatts

“We are seeing wind generation down 60% week-over-week”

https://www.bostonglobe.com/2021/02/16/nation/why-is-power-o...