News for hackers. Stuff that matters.
Re:PG&E turning off power to protect against fire has create... by dredmorbius on Wednesday October 30, 2019 @04:55AM (#21395713)
In many parts of the world, power is unreliable due to a lack of resources -- fuel, capital, labour, distribution, maintenance. Power cuts occur, either scheduled or unscheduled, due to an inability to meet demand.

The case in California is different. The fundamental resources are present: there is fuel (or, in many cases, hydroelectric, geothermal, nuclear, wind, or solar power) for generation. Distribution systems exist. There is no shortage of local labour. Systems aren't going offline due to direct failures of maintenance.

Rather: the outages are prompted by a consequence risk. Given the extreme fire danger throughout the state, the least source of ignition creates a fire hazard, and electrical transmission and distribution equipment is a high risk for such ignition.

Previous major wildfires have been sparked by: mower blades, rims of flat tires sparking on pavement, catalytic converters parked over dry grass, sparks from a hammer. And electrical equipment.

There's a case to be made that PG&E have neglected maintenance activies not related directly to continued operation of their plant, but to risk mitigation, and I'm not going to argue one way or the other.

But the fundamental problem California's electric utilities face is not that of most of the world, where means are insufficient, but that consequence is too great to ignore. I'm coming to see this generally as a late-stage dynamic in many technological areas, one in which the unintended consequences (sometimes unanticipated, often not) simply rise to a level of major significance. You might think of these generally as hygiene factors (the phrase I've been using).

Past examples would include:

- Basic personal and public hygiene factors for maintaining public health in cities.

- Pollution control laws affecting solid, liquid, and gas emissions of activities and plants.

- Environmental contamination as with lead, asbestos, pesticides, tobacco, sulfur dioxide ("acid rain"), and CO2.

- Individual psychological and widespread sociological negative impacts of information overload, media saturation and manipulation, and computer systems overexposure.

- Running extensive high-capacity electrical distribution systems through increasingly fire-prone landscapes.

Or put another way: this is increasingly our future.

Re:PG&E turning off power to protect against fire has create... by jerf on Wednesday October 30, 2019 @09:56AM (#21397751)
"Previous major wildfires have been sparked by: mower blades, rims of flat tires sparking on pavement, catalytic converters parked over dry grass, sparks from a hammer. And electrical equipment."

Honestly, at that point, you have a fire, it's just not burning yet. The idea that we can prevent fires from happening by preventing all sources of ignition in hundreds and hundreds of square miles is completely absurd. Even though Man is responsible for a lot of ignition, he's not responsible for all of them. Even if you completely removed civilization, it's still going to be a fire.

I don't have a solution, or any good news here. Best thing I can think of at this point is spending a shit ton of money and creating some very large fire breaks around cities and then burning on purpose, but California probably can't do that either. By the time the eminent domain lawsuits and counterlawsuits happened, the environmental review happens ("we plan on killing lots of things, and then we're going to set the rest of the stuff we didn't kill on fire" isn't exactly going to sail through the review process), and everyone gets their beaks wetted, it'll just end up being a giant waste of money that gets ended with the fires occur anyhow. And the fire breaks would have to be huge; we're probably measuring the necessary widths with miles fairly reasonably. (Possibly <1 mile, but certainly multiple tenths of a mile.) And the liability. Nobody sane would take that job.

Re:PG&E turning off power to protect against fire has create... by rsync on Wednesday October 30, 2019 @10:09AM (#21397889)
"Even though Man is responsible for a lot of ignition, he's not responsible for all of them. Even if you completely removed civilization, it's still going to be a fire."

Actually, other than a one-in-a-billion naturally occurring compost pile that, by chance, falls into place and reaches burning temperature, internally, the only natural source of wildland fire ignition is lightning.

Which is interesting because in many parts of coastal Marin/Sonoma/Mendocino/Napa counties, there is, essentially, no lightning. I have lived in Marin for ten years and have seen lightning exactly one time - and that was during the rainy season when ignition would have very low consequence.

Wildland fires in the north bay / Sac Delta / Wine Country are ignited by people ...

Re:PG&E turning off power to protect against fire has create... by jerf on Wednesday October 30, 2019 @10:25AM (#21398081)
My core point is that if your "fire prevention" plan is to drive the number of ignition points to zero, you don't have a fire prevention plan, because even the impossible isn't going to be enough. It's not a sensible answer, and it's not really sensible to blame the fires on the individual ignition events.

Or, to put it another way, when we set off an explosive, the explosion isn't really "caused" by the electrical spark; the unusual thing that happened that primarily caused the detonation is the collection of a lot of explosives in one place. Explosives experts are careful around any concentration of explosives, because that's the real cause. By contrast you can play with the electrical detonation system in isolation all day long and at most you'll shock yourself a bit, and that only with extreme carelessness. The primary cause of these fires is the accumulation of huge amounts of easily-burnable material, not which of the thousands+ of ignition events was the unlucky one to set it off.

Re:PG&E turning off power to protect against fire has create... by dredmorbius on Wednesday October 30, 2019 @12:14PM (#21399304)
Human sources are about 85% of wildfire ignitions in California.

That still leaves 15% of ignitions from natural sources.

https://www.latimes.com/california/story/2019-10-29/how-do-w...

Removing the human ignition would likely mean that the blazes from natural sources would take up the slack, and become larger.

Another fact that I'm reminded of was that native Americans would themselves set deliberate blazes which would burn over lands fairly frequently. This would mean that there was a history dating back over centuries, possibly several thousands of years, in which large-scale burns were relatively frequent. This would have phased out beginning with the first European settlements in 1769. By 1900, there had been up to 130 years of diminished fire activity already, not from suppression, but from reduced ignition (though intentional burns by early Europeans were also practiced). Reference with multiple further citations: http://www.californiachaparral.com/enativeamericans.html

The big change seems to be with the 20th century belief that fire is an invader, undesirable, and should be massively suppressed.

Re:PG&E turning off power to protect against fire has create... by nostrademons on Wednesday October 30, 2019 @01:25PM (#21400184)
"The idea that we can prevent fires from happening by preventing all sources of ignition in hundreds and hundreds of square miles is completely absurd."

That's true, but PG&E equipment has started more than 1500 fires between 2014 and 2017. That's over one fire per day. 17 out of the 22 biggest California wildfires in the last 5 years have been started by PG&E:

https://outline.com/5bxFLb

Arguably the power shutoff was the right short-term move here: during this wind event 4 fires have been traced back to PG&E (so far), but they had 50 incidents of damaged or downed power lines. It could've been a lot worse.

Think of it like buffer overflows in C. It's naive to think that you're going to eliminate all security bugs by any one intervention. However, if one class of bugs is responsible for 80% of breaches, isn't it worth investigating to see what you can do about that one class of bug? Here we have an organization that's responsible for 80% of forest fires: you're not going to eliminate all forest fires by firing its management, but shouldn't you at least investigate and see what they might be doing wrong?

Re:PG&E turning off power to protect against fire has create... by jerf on Wednesday October 30, 2019 @01:40PM (#21400348)
"However, if one class of bugs is responsible for 80% of breaches, isn't it worth investigating to see what you can do about that one class of bug?"

Fires become bigger and bigger the more fuel they accumulate. Security vulnerabilities lack this attribute.

Arguably, the best solution wouldn't be to try to extinguish the sources of ignition, but greatly increase them so that it's just something that happens all the time. (Another thing they definitely don't share in common with security vulnerabilities.) Of course, the best solution is to hop in a suitably tricked-out De Lorean and start that policy a hundred years ago.

The steady-state of this plan is probably pretty nice. However, getting there from here is a real problem. But I'm not sure there's any way around that problem anymore. That's another part of what I mean by "there's already a fire, it just isn't burning yet"; trying to pretend that there's some solution that doesn't involve fire is just fooling oneself. It's only a question of when, not if.

If California could be trusted to do something useful with the time, enough so that they get meaningfully ahead of the still-accumulating fuel, "buying time" would be a viable option, but I see very little evidence that anything useful is being done. Links solicited to the contrary; I haven't gone looking. But if the solution is just "try to never ever spark anything and thus The Problem Is Solved!", yeah, that's not a solution. It does nothing to stop the fire-in-waiting from getting bigger.

Re:PG&E turning off power to protect against fire has create... by dredmorbius on Wednesday October 30, 2019 @01:51PM (#21400476)
One nit on an otherwise strong explanation:

Fires become bigger and bigger the more fuel they accumulate. Security vulnerabilities lack this attribute.

I'd argue that this isn't the case. The threats of Big Data and Big Data Breaches is directly analagous to fuel-load accumulation, with the added bonus of attractive nuisance / induced risk.

That is, we're seeing the same class of bugs we've always seen (ignition sources), but the consequences are vastly higher (international finance, national security, organised crime, personal details and blackmail on billions of souls, greatly expanded venues for long-standing frauds).

Buffer overflows, viruses, worms, trojans, social engineering, phishing, and MITM all predated the past decade. Advanced-fee fraud, impersonation, Ponzi schemes, etc., all existed previously. Married through high-speed, high-capacity data networks, tied to comprehensive and deep data stores, the whole system is weaponised. And the stakes and consequences are huge.

Rather like wildfires in California.

By the way, I've got a Mr. Fusion cannister sitting in the garage, interested?

Re:PG&E turning off power to protect against fire has create... by jerf on Wednesday October 30, 2019 @01:53PM (#21400503)
Fair enough... I was speaking in a fairly limited sense where having one security vulnerability in a product doesn't necessarily impact another (assuming they can't be chained) on a single product, but there are dimensions in which it is similar, yes.
Re:PG&E turning off power to protect against fire has create... by dredmorbius on Wednesday October 30, 2019 @05:15PM (#21402823)
Thinking cross-domain on various threats and risks is interesting and useful. I wouldn't have thought of the situation if you'd not made your assertion, so it was useful in that way ;-)
Re:PG&E turning off power to protect against fire has create... by nostrademons on Wednesday October 30, 2019 @01:56PM (#21400536)
> Fires become bigger and bigger the more fuel they accumulate. Security vulnerabilities lack this attribute.

Security vulnerabilities do have that property: if you find that buffer overflow, you can usually get an RCE, which will let you get root, which will let you compromise the network that the box is on, which will let you find additional vulnerabilities (human or otherwise) to compromise other systems. That's why competent security teams practice defense in depth, and have internal trip-wires and firewalls to limit the damage of any intrusion. That doesn't absolve you from trying to stop the intrusion in the first place.

Similarly, getting PG&E to maintain their equipment is not mutually exclusive with other fire mitigation strategies like controlled burns, defensible space, homeowner education, etc. If anything it'll free up resources for those interventions by switching fire departments out of, well, firefighting mode and giving them some breathing space to focus on more long-term practices.

Re:PG&E turning off power to protect against fire has create... by dredmorbius on Wednesday October 30, 2019 @11:07AM (#21398542)
The point that any ignition source no matter how small can spark a major wildfire is precisely my point. I don't think you're contesting this, but I want to clarify the agreement.

As to clearing and preemptively burning regions ... one of the more terrifying items that I ran across in the past day was that the Kinkaid fire was burning through areas in which a prior burn, 2017's Tubbs fire, had already burned out:

https://sfist.com/2019/10/28/kincade-fire-doubles-in-size-tu...

It's not clear how thorough the Tubbs fire was in those areas, or if spread of the Kinkaid fire was moderated. But the fact that two major wildfires can burn through the same area only two years apart is ... sobering.

What's effectively NIMBY opposition to wider prescribed burns, firebreaks, expanded evacuation routes, evacuee housing/facilities, structure code and inspection requirements, etc., does seem to be a major factor.

But that's still not addressing the fact that fires are simply bigger, hotter, faster, and more unpredictable than they were even just 20 years ago.

Re:PG&E turning off power to protect against fire has create... by throw0101a on Wednesday October 30, 2019 @08:16AM (#21396820)
> Previous major wildfires have been sparked by: mower blades, rims of flat tires sparking on pavement, catalytic converters parked over dry grass, sparks from a hammer. And electrical equipment.

A citation:

* https://www.latimes.com/california/story/2019-10-29/how-do-w...

Another one:

> Fear of insects. A rancher tried to plug a wasp’s nest in the ground by jamming a stake into the ground. That unleashed a spark that began burning waist-high grass. He tried to smother the flames by tossing a trampoline on it, but that just worsened the flames. It caused the start of the largest of two fires that merged to become California’s largest wildfire on record, the Mendocino Complex fire, which burned more than 450,000 acres in four Northern California counties — Colusa, Lake, Mendocino and Glenn.

The above was Mendocino:

* https://en.wikipedia.org/wiki/Mendocino_Complex_Fire

Re:PG&E turning off power to protect against fire has create... by dredmorbius on Wednesday October 30, 2019 @11:19AM (#21398673)
Thanks. I should have supplied cites, wasn't feeling up to it.

About three weeks ago: https://news.ycombinator.com/item?id=21218868

Citing: https://www.nytimes.com/2018/08/20/us/california-wildfires-h...

Re:PG&E turning off power to protect against fire has create... by hnuser77 on Wednesday October 30, 2019 @11:10AM (#21398577)
My understanding was that the main reason power has to be turned off because there could be 1) vegetation contact with the power lines in windy conditions 2) power lines blowing away in the wind. California has been windy since before the invention of electricity, so what's changed? They can't build power lines to handle some wind that happens every other year since the beginning of human settlement? If they are failing to trim and remove dying trees around power lines, that is a direct maintenance failure. Besides PG&E, at the forestry management level, not clearing brush or doing controlled burns is direct maintenance failure.
Re:PG&E turning off power to protect against fire has create... by djokkataja on Wednesday October 30, 2019 @12:59PM (#21399883)
> I'm coming to see this generally as a late-stage dynamic in many technological areas, one in which the unintended consequences (sometimes unanticipated, often not) simply rise to a level of major significance. You might think of these generally as hygiene factors (the phrase I've been using).

You mean like externalities? https://en.m.wikipedia.org/wiki/Externality

> Or put another way: this is increasingly our future.

This seems like a situation that might not be best served by a startup... So yes, this is probably what California's future looks like.

Re:PG&E turning off power to protect against fire has create... by PeterStuer on Wednesday October 30, 2019 @06:09AM (#21396058)
You could argue that the lack of resources also plays in California, not because of a fundamental shortfall, but due to all the resources being sucked out of the system by extreme capitalism.
Re:PG&E turning off power to protect against fire has create... by dredmorbius on Wednesday October 30, 2019 @11:20AM (#21398678)
Howso?

Resources devoted to firefighting have increased by any possible measure.