News for hackers. Stuff that matters.
Re:Cloudflare API, dashboard, tunnels down by dijit on Wednesday October 25, 2023 @01:12PM (#38015213)
If you’re waiting for the inevitable “this is why we shouldn't centralize everything”; this is that comment.

As much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope.

luckily this time its just administrative interfaces and supplemental services that assist in development and that hopefully aren't used in production.

People will defend cloudflare here with the old saying “Operating at scale is hard” but simultaneously holding the opinion that operating services themselves is equally hard.

I guess the argument is that outsourcing is easier, but then you are very much out of control.

“the internet” in its practical entirety going down becomes more possible every passing day.

Re:Cloudflare API, dashboard, tunnels down by cj on Wednesday October 25, 2023 @01:40PM (#38015658)
Not to take away from your point, but the centralization around AWS and GCP and Azure is more concerning to me than the centralization around Cloudflare.

Multi-cloud deployments aren’t the norm, so most companies are centralized on 1 cloud provider.

If we go even deeper, the entire internet runs on a limited number of undersea cables connecting continents. That in particular has always seemed like the most vulnerable aspect of the internet, considering a targeted attack on < 100 locations could meaningfully cut off internet access for millions (billions?) of people for an extended period of time.

Re:Cloudflare API, dashboard, tunnels down by jabart on Wednesday October 25, 2023 @02:07PM (#38016067)
This. If downtime is that big of an issue, you should be on Cloudflare's business/enterprise plan which allows CNAME DNS, have redundant DNS providers with 30s TTLs that flip over to another provider. Better yet if you can 50/50 the traffic and 100/0 it when something goes bad. Downside is cost and DNSSEC has an issue with multiple keys. I think there was an RFC on that though I'm not sure it's well supported yet.
Re:Cloudflare API, dashboard, tunnels down by pixl97 on Wednesday October 25, 2023 @02:29PM (#38016397)
>then deploy and split load between 2 or more providers (e.g. Cloudflare, Fastly)

How does this work when most edge providers manage your DNS to point at the available servers they have?

Also, if you "HAVE" to be online, then you're the type of target that's at big risk of DDOS which complicates this even further.

Re:Cloudflare API, dashboard, tunnels down by wbl on Wednesday October 25, 2023 @05:29PM (#38018638)
Usually each one will give you a CBAME, your DNS or Cedexic decides, sometimes per client. It's a pretty common setup although can complicate config changes and taking advantage of new features.
Re:Cloudflare API, dashboard, tunnels down by solardev on Wednesday October 25, 2023 @01:35PM (#38015572)
Is it really that bad? Services go down for a few minutes, maybe a few hours? A thousand other companies are down at the same time.

Grab lunch, come back, and it'll have fixed itself. Nobody blames you because it's just another provider outage -- whether it's Cloudflare or AWS or Microsoft or Gmail doesn't really matter, it's a convenient outsourcing of not just network admin but also blame.

The alternative is, what, doing it all in-house with a bunch of people on pager duty? Few companies can afford that. Diversifying across multiple cloud providers? Sure, if multi-nines uptime is really critical for your business.

For most small-med businesses, it's more valuable to be able to rapidly build on top of existing, professionally-maintained infrastructure in order to deliver some unique business value on top of it, than to recreate that basic network infra themselves. If you're not a FAANG, going down for a few hours a year isn't really a big deal. Especially when you can just blame it on an overall "internet outage at some big provider" rather than some internal fuck-up.

Re:Cloudflare API, dashboard, tunnels down by csomar on Wednesday October 25, 2023 @01:47PM (#38015764)
I don't think the problem is a few minutes of down time but all services going down at the same time. You can't order food (Uber down), your email down, your TV down, maybe even the traffic control?
Re:Cloudflare API, dashboard, tunnels down by solardev on Wednesday October 25, 2023 @01:48PM (#38015777)
Ok, and...?

(Edit: Not to be snarky, but really, so what if that happens? Same thing happens if there's a power outage in your neighborhood. Is it a big deal?)

Re:Cloudflare API, dashboard, tunnels down by king_phil on Wednesday October 25, 2023 @02:29PM (#38016399)
Are power outages even a thing? My PC at home (germany) is always on, so I'd notice, and I'm involved with energy at our data center (Frankfurt, Germany) and home and external data center had no outage for the last 20 years, at a minimum.
Re:Cloudflare API, dashboard, tunnels down by solardev on Wednesday October 25, 2023 @02:34PM (#38016481)
Heh, you gotta remember that the US is a third-world country in comparison to much of Europe. In the American West, wildfires frequently bring down our grids, often for hours or days at a time. Many of our power lines are above ground, and when they burn down, thousands of homes and businesses will lose power. And in big storms, sometimes the power lines will start fires themselves. We have started resorting to planned rotating blackouts preemptively, and some areas have a vague plan to upgrade their grids over decades, but AFAIK I don't think there is a concerted effort to really deal with this meaningfully and quickly enough.

It's going to get worse, too, with climate change compounding with increased wildfires compounding with increased electric vehicle uptake compounding with an aging grid and ancient infrastructure compounding with the irregular generation of renewables compounding with an ever-more-broken government compounding with bankrupt utilities. It's not a pretty picture of our future energy stability :)

Edit: It's another vote for cloud services, IMO, that often have multiple power inputs, backup power and generators, cross-regional availability, etc.

Re:Cloudflare API, dashboard, tunnels down by seabrookmx on Wednesday October 25, 2023 @03:15PM (#38016985)
Yes, rural communities exist.
Re:Cloudflare API, dashboard, tunnels down by pinkgolem on Wednesday October 25, 2023 @04:39PM (#38018057)
Germany has an average of 30 minutes of downtime per person per year, Amerika seems to be around 8 hours

Could not find data how this is distributed..

Re:Cloudflare API, dashboard, tunnels down by pixl97 on Wednesday October 25, 2023 @02:24PM (#38016325)
>Grab lunch

Sorry man, our Stripe terminals aren't working, we can't even accept cash... Um, come back later.

Re:Cloudflare API, dashboard, tunnels down by solardev on Wednesday October 25, 2023 @02:32PM (#38016449)
That actually happens with some frequency where I live, and they either use the old-school card imprint machines, or take down your number manually, or some terminals have an offline mode where it'll save the cards and then batch-run them later. It's also rare enough that it's not an issue at all in regular life.
Re:Cloudflare API, dashboard, tunnels down by ta1243 on Wednesday October 25, 2023 @01:17PM (#38015313)
Operating at scale is hard.

However the internet doesn't require people to "operate at scale". That's the beauty. It's a mash of different (relatively) small autonomous systems, with many hosts. If any host goes down, or any network goes down, the system ignores it, or reroutes it.

Re:Cloudflare API, dashboard, tunnels down by baq on Wednesday October 25, 2023 @01:20PM (#38015350)
In practice, cloudflare or AWS down means half the internet doesn't work at all. The half that works - routing around problematic networks - isn't very helpful when the problematic networks are the useful ones.
Re:Cloudflare API, dashboard, tunnels down by ta1243 on Wednesday October 25, 2023 @01:26PM (#38015458)
Depends what you want to get to. HN doesn't use any centralised services.

If someone chooses to host their service relying on a single unreliable point of failure that's on them. Plenty more hosts on the web.

Re:Cloudflare API, dashboard, tunnels down by solardev on Wednesday October 25, 2023 @01:37PM (#38015610)
> Depends what you want to get to. HN doesn't use any centralised services.

It's also one of the least reliable and performant public forums I've ever used. It goes down or slows down several times a week for me. Wish they'd just move it to the cloud instead of occasionally upgrading the bare metal...

Re:Cloudflare API, dashboard, tunnels down by inemesitaffia on Thursday October 26, 2023 @04:31AM (#38023165)
They used AWS when they had lost their SSD's
Re:Cloudflare API, dashboard, tunnels down by djbusby on Wednesday October 25, 2023 @01:32PM (#38015542)
Isn't HN centralized on like two servers in CA?
Re:Cloudflare API, dashboard, tunnels down by shanemhansen on Wednesday October 25, 2023 @01:55PM (#38015881)
Well technically they do, but honestly the number of companies that need to worry about redundant DNS is probably .001%. It does matter for the some people, but as someone from SRE once said "100% availability is the wrong target for basically everything".

dig SOA news.ycombinator.com

; <<>> DiG 9.18.16-1-Debian <<>> SOA news.ycombinator.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10818 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;news.ycombinator.com. IN SOA

;; AUTHORITY SECTION: ycombinator.com. 900 IN SOA ns-225.awsdns-28.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 39 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Wed Oct 25 10:52

Re:Cloudflare API, dashboard, tunnels down by mschuster91 on Wednesday October 25, 2023 @01:27PM (#38015468)
> However the internet doesn't require people to "operate at scale".

The internet itself does not, yes, but the internet as a system, filled to the brim with bad actors ranging from 14 year old skiddies with Metasploit over hardened Russian ransomware groups to virtually all nations' secret services and militaries does.

As soon as you start a server with a globally routed IP address, it will get hammered from all kinds of scanners. Anything on port 80/443 will get bombarded with Wordpress, Drupal and log4j attempts, anything on port 22 will get hounded by people attempting credential stuffing, and anything on port 25/587 by spammers and anyone trying to exploit Exchange/Exim vulnerabilities. It's a nightmare to keep up with that crap alone, and you're one forgotten update away from getting your server 0wned by someone using Shodan and whatnot.

And then, once your site, whatever you operate - I've seen as an admin DDoS extortion attacks happen against small sites for dog walker services, pizza parlors or random bloggers as well as large car dealerships - finds itself some opponent for whatever reason, it gets nasty. Be it for political reasons (people don't like what the blogger writes), for the lulz (people don't like a Twitch streamer so they DDoS their sponsors), or for financial extortion, doesn't matter in a time where you can order such attacks for less than the price of your average McDonalds visit. There is no, utterly no, alternative to the giants (Cloudflare, AWS CloudFront, Akamai).

It's sad that it has come to that point, but unless we get the most obnoxious sources of such attacks - enemy nation states (China, Russia) and compromised domestic IoT devices/servers - off of the Internet, there is really no viable alternative.

Re:Cloudflare API, dashboard, tunnels down by ta1243 on Thursday October 26, 2023 @07:45AM (#38024299)
> There is no, utterly no, alternative to the giants (Cloudflare, AWS CloudFront, Akamai).

Yet my servers are fine without such services.

Re:Cloudflare API, dashboard, tunnels down by shepherdjerred on Wednesday October 25, 2023 @01:20PM (#38015357)
What's the impact for most sites if they go down a few times a year due to AWS/CloudFlare?
Re:Cloudflare API, dashboard, tunnels down by dijit on Wednesday October 25, 2023 @01:23PM (#38015406)
Whats the impact if the entire internet is down for a couple of hours?

What if its more than a few hours because theres a natural monopoly and a dearth of capable people to run systems independently - thus no strong external incentive for rapid response outside of punitive SLAs?

How much soft power do you think such an entity would have on the world stage.

Re:Cloudflare API, dashboard, tunnels down by solardev on Wednesday October 25, 2023 @01:44PM (#38015709)
> Whats the impact if the entire internet is down for a couple of hours?

People get some off-screen time? Life goes on.

> What if its more than a few hours because theres a natural monopoly?[...snip...] How much soft power do you think such an entity would have on the world stage.

They're just commodity infra providers. The Facebook and Twitters of the world have way more power and influence. This isn't even a hypothetical. The big cloud providers have been around for what, a decade or so now? How many times has the world collapsed with every Cloudflare or AWS downtime? It's just not as big a deal as you paint it to be. These providers going down is like the digital equivalent of a highway crash -- a small inconvenience for a few hours, then forgotten about in a day or two.

Re:Cloudflare API, dashboard, tunnels down by shepherdjerred on Wednesday October 25, 2023 @05:05PM (#38018369)
> Whats the impact if the entire internet is down for a couple of hours?

Okay, so obviously a lot of "important" stuff happens via the Internet. You could come up with some financial impact if the Internet went down for some time.

Harder to measure: A lot of people would be inconvenienced or suffer. Some number of people would die indirectly, e.g. due to emergencies/lack of communication.

People are inconvenienced and suffer and die all of the time. It's part of life.

I'm not advocating for more death and suffering, but I am saying that, as another comment said, "life goes on".

---

Put another way, if the Internet disappeared for a week, month, year, what would happen? It'd suck and we'd adjust, but we (humanity as a whole) would be fine.

Re:Cloudflare API, dashboard, tunnels down by rafaelmn on Wednesday October 25, 2023 @01:33PM (#38015552)
If you think internet infrastructure is centralized, boy are you going to be depressed about AI powered services.

Those are much more volatile (compute heavy and not enough compute to meet peaks), harder to setup/maintain and benefit greatly from sharing infrastructure for hardware utilization. Everything is moving in that direction.

Re:Cloudflare API, dashboard, tunnels down by SheinhardtWigCo on Wednesday October 25, 2023 @01:31PM (#38015530)
The current situation is preferable to the before times, where any website could be nuked from orbit by anyone with a mild grievance. Cloudflare has succeeded in effectively shutting down that attack vector, and that’s worth the risk of occasional brief but broad outages, at least to me.
Re:Cloudflare API, dashboard, tunnels down by dijit on Wednesday October 25, 2023 @02:01PM (#38015976)
There were mitigations for this but it was usually on “IT”s budget and IT was considered a cost center.

Really the best innovation in the last 10 years has been the penchant for putting things on development budgets instead which loosened the purse strings considerably.

I wish I had sufficient data to back this up but for context; I was part of a small infrastructure team running a little over 1% of all web traffic at one point in a pair of colos (with DDoS protection) for less than £5MM on infra costs.

Now I can easily be in the 6 digits with just a CRUD app

Re:Cloudflare API, dashboard, tunnels down by pvg on Wednesday October 25, 2023 @01:33PM (#38015556)
the inevitable

It's readily evitable, just choose not to write the generic, repetitive thing you already know is generic and repetitive.

Re:Cloudflare API, dashboard, tunnels down by gjsman-1000 on Wednesday October 25, 2023 @01:24PM (#38015422)
Most people blame Cloudflare and Amazon for centralizing everything.

I actually disagree; the centralization is a symptom of severe, systemic, unfixable flaws in the design of the internet itself. At least we didn't let AOL define internet.

Re:Cloudflare API, dashboard, tunnels down by justin_oaks on Wednesday October 25, 2023 @01:39PM (#38015633)
I'd appreciate if you'd elaborate on that. (Edit: I see you elaborated somewhat in another comment posted before I finished this comment)

I tend to think centralization is more about people's tendencies than it is internet design. As I see it, centralization on the internet happens as the emergent behavior of all of the elements in the system and the forces that exist. In other words, it's the natural result of incentives.

People use centralized services because

- prices (economies of scale)

- network effects ("I can't certain people without WhatsApp")

- social pressures ("People look at me weird when I say I don't have an Instagram")

- visibility ("There are other CRMs than SalesForce?")

- making choices easy ("I don't want to choose between all these options. I'll pick the most familiar")

- safety ("Nobody got fired for choosing IBM")

- convenience ("It's a pain to self-host")

- etc

Re:Cloudflare API, dashboard, tunnels down by gjsman-1000 on Wednesday October 25, 2023 @01:46PM (#38015747)
True; but let's imagine everyone self-hosted. Perfectly decentralized. As I edited another comment to reflect, imagine if you took out just three companies.

Level 3 Communications / Lumen; AT&T; Verizon Enterprise.

With those three, you've knocked 2.35 million km, or 1.46 million miles, of fiber optic offline. That's over half of the US backbone in those three companies. Comcast "only" has 0.15 million miles for the sake of comparison. The internet would crash and burn immediately to speeds not seen since the dial-up era. There's nothing we can do about that. Cloudflare and AWS centralizing everything is actually, statistically, less risky than backbone attacks. If you succeeded in knocking down all of the big seven at once; you've got about 80-90% of the backbone offline plus all of the online cloud providers with direct connections to them; at which point the internet would cease to function in any practical sense.

Re:Cloudflare API, dashboard, tunnels down by DoctorOW on Wednesday October 25, 2023 @03:43PM (#38017303)
I don't think this is more centralized, because it requires taking out multiple companies services. To take down three of these networking companies at once is something we've never actually seen, and that's the level needed to be comparable to a CloudFlare failure.
Re:Cloudflare API, dashboard, tunnels down by wbl on Wednesday October 25, 2023 @05:30PM (#38018651)
Bad big in transitive bgp data parsing will take out all networking from one vendor.
Re:Cloudflare API, dashboard, tunnels down by djbusby on Wednesday October 25, 2023 @01:33PM (#38015557)
Such as?
Re:Cloudflare API, dashboard, tunnels down by gjsman-1000 on Wednesday October 25, 2023 @01:37PM (#38015612)
1. DDoS Attacks

2. Port scanning attacks

3. Geolocation attacks (you shouldn't run a server from your home, when your IP Address could be used to find you; be it from a hack or an abuse of the legal process)

4. The US internet relying on a backbone mostly operated by 4 companies (meaning a direct connection is a massive perk)

For as much as we moan about Amazon and Cloudflare centralizing things; believe me, just taking out Level 3 / Lumen would be almost an act of war and would affect every ISP and cloud provider.

If you took out Lumen, Verizon Enterprise Solutions, and AT&T; that's 2.35 million kilometers of fiber going dark. For reference, Comcast has "only" 150,000 km.

Re:Cloudflare API, dashboard, tunnels down by djbusby on Wednesday October 25, 2023 @01:41PM (#38015665)
Those are problem and symptoms - not design flaws. How could one design a system that doesn't let DDoS exist on the network?
Re:Cloudflare API, dashboard, tunnels down by silisili on Wednesday October 25, 2023 @01:46PM (#38015743)
I actually have a lot of respect for the big backbones. If Goog/MS/Amz owned them all, they'd find some way to wall it off from everyone else. If Comcast or Charter owned them all, our internet would cost about $1000 a month by now. All things considered, IMO they've been fairly good and fair stewards.
Re:Cloudflare API, dashboard, tunnels down by swader999 on Wednesday October 25, 2023 @01:43PM (#38015692)
It's nice to at least be considering blast radius in a figurative manner, hope that lasts.