News for hackers. Stuff that matters.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by divbzero on Saturday November 04, 2023 @02:49AM (#38138742)
Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon. The three data centers are independent of one another, each have multiple utility power feeds, and each have multiple redundant and independent network connections. The facilities were intentionally chosen to be at a distance apart that would minimize the chances that a natural disaster would cause all three to be impacted, while still close enough that they could all run active-active redundant data clusters.

If the three data centers are all around Hillsboro, Oregon, an earthquake could probably take out all three simultaneously.

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by pests on Saturday November 04, 2023 @03:13AM (#38138831)
> Hillsboro, Oregon, an earthquake could probably take out all three simultaneously.

Is it west of I5?

(yes)

Oh yeah, they all gone.

Cascadia Subduction Zone - https://pnsn.org/outreach/earthquakesources/csz

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by bell-cot on Saturday November 04, 2023 @06:57AM (#38139857)
Wikipedia's entry for Hillsboro: "Elevation 194 ft (60 m)"

Between that, and being ~50 miles inland - I'd say there's ~zero threat of Cascadia quakes or tsunamis directly knocking out those DC's. (Yeah, larger-scale infrastructure and social order could still be killers.)

OTOH - Mt. St. Helens is about 60 miles NNE of Hillsboro. If that really went boom, and the wind was right...how many cm's of dry volcanic ash can the roofs of those DC's bear? What if rain wets that ash? How about their HVAC systems' filters?

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by pests on Saturday November 04, 2023 @08:57AM (#38140637)
I was never worried about the tsunami. Okay, maybe not gone, but I wouldn't say it would be operational.

https://www.oregon.gov/oem/Documents/Cascadia_Rising_Exercis...

50% of roads and near 75% of bridges damaged on the west coast and the I5 corridor.

Refer to PDF page #93 where over 70% of power generation is highly damaged on the I5 corridor and 60% in the coastal areas with 0% undamaged.

Highly damaged - "Extensive damage to generation plants, substations, and buildings. Repairs are needed to regain functionality. Restoring power to meet 90% of demand may take months to one year."

"In the immediate aftermath of the earthquake, cities within 100 miles of the Pacific coastline may experience partial or complete blackout. Seventy percent of the electric facilities in the I-5 corridor may suffer considerable damage to generation plants, and many distribution circuits and substations may fail, resulting in a loss of over half of the systems load capacity (see Table 22). Most electrical power assets on the coast may suffer damage severe enough as to render the equipment and structures irreparable"

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by bell-cot on Saturday November 04, 2023 @09:56AM (#38141024)
Good backups generators at their colo's could handle the lack of utility power for days to weeks. More & better generators could be hauled in and connected.*

The two big problems I'd see would be (1) Social Order and (2) Internet Connectivity. DC's are not fortresses, and internet backbone fibers/routers/etc. are distributed & kinda fragile.

*After all the large-scale power outages & near-outages of recent decades, Cloudflare has no excuse if they lack really-good backup generators at critical facilities. And with their size, Cloudflare must support enough "critical during major disaster" internet services to actually get such generators.

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by ZiiS on Saturday November 04, 2023 @07:01AM (#38139882)
And most of thier SREs. Spending 30 hours to recover from the worst natural disaster in recorded history is slightly diffrent then from a ground fault on a single transformer.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by fanf2 on Saturday November 04, 2023 @01:28PM (#38143192)
There’s also the Portland Hills and Gales Creek fault zones to worry about.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by nanankcornering on Saturday November 04, 2023 @02:57AM (#38138762)
thats why DRC exists right?
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by ahoka on Saturday November 04, 2023 @05:22AM (#38139359)
They would then go into very detailed description how tectonic activity caused the outage.