News for hackers. Stuff that matters.
Re:Cloudflare had a partial outage by phillipseamore on Tuesday June 21, 2022 @03:19AM (#31821154)
It's time to start discussing a fail-open option for us CF users. Most of my sites are using CF for global performance rather than DDoS protection and security. I'd be fine with them changing DNS to point to the origin (or any other user defined IPs) in case of issues (even if it would take hours to return to normal).

This is also important for countries with limited connectivity to the Internet, if the PoP in that country looses it's connection back to CF it shuts everything down, so even if the origin is in the next rack over from the PoP, it's un-reachable.

Re:Cloudflare had a partial outage by renonce on Tuesday June 21, 2022 @03:35AM (#31821277)
You can implement your own DNS server that CNAMEs to Cloudflare and falls back to origin IP when there is a problem with Cloudflare. I think a downstream Cloudflare provider could provide such services if they desire.
Re:Cloudflare had a partial outage by subvarad on Tuesday June 21, 2022 @03:44AM (#31821339)
Outages occur to the best of CDNs. But the likelihood of two CDNs suffering an outage at the same time is close to zero. Check out www.cdnreserve.com.
Re:Cloudflare had a partial outage by phillipseamore on Tuesday June 21, 2022 @03:36AM (#31821282)
Last time I checked that was limited to the enterprise plan.
Re:Cloudflare had a partial outage by eins1234 on Tuesday June 21, 2022 @04:28AM (#31821642)
Of course it is... Typical Cloudflare.
Re:Cloudflare had a partial outage by Cthulhu_ on Tuesday June 21, 2022 @04:33AM (#31821677)
Imagine having to pay for a service
Re:Cloudflare had a partial outage by adwww on Tuesday June 21, 2022 @04:44AM (#31821742)
Cloudflare's pricing has issues.

My company was paying $20 a month. We were heavily depended on CF, we'd have been happy to pay more.

But... the one feature we wanted was for our accounts team to have their own login so the ops team didn't have to download invoices every month. Nope, that one feature required an enterprise plan which they quoted $4,000 a month for.

Re:Cloudflare had a partial outage by garyclarke27 on Tuesday June 21, 2022 @05:17AM (#31821945)
Oh dear your poor Ops team, they had to download a few invoices to save $4k and every month! My heart bleeds for them.
Re:Cloudflare had a partial outage by tinco on Tuesday June 21, 2022 @05:52AM (#31822149)
Companies where you have to log in and download invoices are the worst. If there's a viable alternative to their products I switch immediately. You make it seem like it's not a big deal, but a reasonably sized startup has dozens of service providers. Should we pay every little service $4k/mo just to save the communications and context switching overhead?
Re:Cloudflare had a partial outage by adwww on Tuesday June 21, 2022 @07:19AM (#31822618)
You jest, but imagine how time consuming it would be if every app we used was setup like CloudFlare, where only the one super admin can deal with billing.

Also in these days of remote work, it's a problem if the credit card details need updating - either you have to give the company card details over a slack call, or you need to give a card holder your root password.

Re:Cloudflare had a partial outage by hizanberg on Tuesday June 21, 2022 @05:10AM (#31821891)
I've been impressed with Cloudflare's (non enterprise) value thus far, what Bandwidth & Users did the $4k quote cover?
Re:Cloudflare had a partial outage by evrflx on Tuesday June 21, 2022 @05:11AM (#31821903)
Is there no api for that? It would even save the manual download effort.
Re:Cloudflare had a partial outage by aaaaaaaaaaab on Tuesday June 21, 2022 @10:59AM (#31824699)
Just write a script?
Re:Cloudflare had a partial outage by mpol on Tuesday June 21, 2022 @04:41AM (#31821724)
Imagine having to pay for a service... at enterprise prices.

This is not really an extra or a nice-to-have, it might be more a hostage situation.

Re:Cloudflare had a partial outage by eins1234 on Tuesday June 21, 2022 @05:05AM (#31821868)
Imagine already paying for a service and then having someone snark at you for wanting things for free.

I tried to exercise some restraint this time, but screw it. Here's another rant:

Beware of Cloudflare's tactic of luring people in to their CDN product with "free" bandwidth, and then locking useful features arbitrarily behind what I can only imagine is a thousands of dollars per month enterprise plan. Just look at their cache-purging page for a super obvious example of this (there are plenty more, way too many to list), everything other than basic purge by URL is enterprise only: https://developers.cloudflare.com/cache/how-to/purge-cache/

These days Cloudflare is literally my last choice for a CDN for my new projects. My new go-to is bunny.net, who charges a reasonable usage-based fee for bandwidth and gives you unfettered access to all the features they've built (and doesn't route your users to farther/closer nodes based on how much you pay: https://cloudflare-test.judge.sh/). Though I'd even reach for Cloudfront with their expensive bandwidth costs these days, because at least their pricing is transparent and scales smoothly with usage, and they don't arbitrarily cut you off from useful features that you might not know you need yet.

Even their bandwidth might not really be "free", since I've heard if you actually use any significant amount, the sales people will come knocking on your door to coerce you to get on the same enterprise plan or have your site taken down.

Re:Cloudflare had a partial outage by easytiger on Tuesday June 21, 2022 @05:48AM (#31822124)
Can I ask out of interest (most of my projects are high perf/low traffic) what kind of traffic you are dealing with at the point you decide you need a CDN?
Re:Cloudflare had a partial outage by eins1234 on Tuesday June 21, 2022 @06:11AM (#31822238)
I don't really use a CDN to manage high traffic volumes. It's more to provide a better, lower-latency experience for my users regardless of where they access my apps from.
Re:Cloudflare had a partial outage by bluejekyll on Tuesday June 21, 2022 @03:27AM (#31821215)
You’d need to have TLS certs on origin ready to go for this scenario to work. Additionally, you’d need to make sure to test it and ensure that there’s nothing wrong in this event.

On top of that, depending on your scale, can you take all the traffic on origin that Cloudflare currently offloads?

Re:Cloudflare had a partial outage by phillipseamore on Tuesday June 21, 2022 @03:29AM (#31821234)
No issue for me. This is obviously a power-user option. It's kind-of implemented for Enterprise users were you don't have to let CF have full control over the domain.
Re:Cloudflare had a partial outage by yunohn on Tuesday June 21, 2022 @04:27AM (#31821637)
> I'd be fine with them changing

This would be best implemented by you. If the point is to avoid CF as a PoF, why would you rely on their infra to fail safe when something breaks?

Re:Cloudflare had a partial outage by booi on Tuesday June 21, 2022 @03:24AM (#31821196)
Wouldn't this expose the origin IPs to attack?
Re:Cloudflare had a partial outage by mccorrinall on Tuesday June 21, 2022 @03:29AM (#31821233)
Yes, but he says in his second sentence that he doesn’t mind and mainly uses CF for performance.
Re:Cloudflare had a partial outage by phillipseamore on Tuesday June 21, 2022 @03:31AM (#31821241)
Plenty of commenters also seem to miss the most important word in the first sentence: "option".
Re:Cloudflare had a partial outage by phillipseamore on Tuesday June 21, 2022 @03:26AM (#31821206)
I'm talking about this as an option for users like me, that don't have an attack surface, but need the global performance gains of CF.
Re:Cloudflare had a partial outage by 1123581321 on Tuesday June 21, 2022 @03:29AM (#31821231)
Probably not many users who need the performance and can handle unexpected failover. There would also be the issue of setting the policy defaults effectively. Most users wouldn’t benefit from this footgun.

If you’re serious, you could probably automate this right now with your DNS provider and uptime monitoring.

Re:Cloudflare had a partial outage by phillipseamore on Tuesday June 21, 2022 @03:34AM (#31821264)
I'm so serious that I already have failover after 1 hour at the registrar level, but those changes are not immediate and can take up to 24h to roll-in and roll-back due to DNS propagation and caching.
Re:Cloudflare had a partial outage by phillipseamore on Tuesday June 21, 2022 @03:45AM (#31821344)
We seem to have hit the nesting maximum 1123581321, but to your point "There is no immediate option with DNS changes."

There's a huge difference in changing nameservers for a domain and simply changing host records.

Re:Cloudflare had a partial outage by efreak on Tuesday June 21, 2022 @04:45AM (#31821745)
This is a 7th level comment. Synu below[0] has a 9th level comment, and I've seen nested comments go quite a bit further.

[0] https://news.ycombinator.com/item?id=31821497

Re:Cloudflare had a partial outage by pxeger1 on Tuesday June 21, 2022 @06:07AM (#31822220)
I think there is no nesting maximum (or if there is, it's much bigger than this). There's a limit which stops you replying to a comment immediately, to prevent super long quick-fire arguments.
Re:Cloudflare had a partial outage by 1123581321 on Tuesday June 21, 2022 @09:40AM (#31823815)
Ah, sorry, misunderstood you. You can’t rely on them to change their host records when they’re down.

If you want CDN-independent automatic failover, look into anycast with two providers. If one of them is Cloudflare, use the tier that lets you manage your DNS elsewhere.

Re:Cloudflare had a partial outage by 1123581321 on Tuesday June 21, 2022 @03:39AM (#31821303)
There is no immediate option with DNS changes. CF can’t immediately remove their IP from the route. Sounds like you’ve solved your problem in the sense that you have an automatic failover, though, which is good.
Re:Cloudflare had a partial outage by chrisweekly on Tuesday June 21, 2022 @07:10AM (#31822565)
Typically CF _is_ the DNS provider though, right?
Re:Cloudflare had a partial outage by 1123581321 on Tuesday June 21, 2022 @10:25AM (#31824291)
Only up until the $200/mo tier. This kind of feature would be locked at that level anyway.

As I said to the OP elsewhere, he should be doing something like anycast to multiple CDNs if this is critical.

Re:Cloudflare had a partial outage by hatthew on Tuesday June 21, 2022 @03:28AM (#31821221)
Yes, but I'd wager that most sites experience cloudflare outages more often than they experience bona fide attacks.
Re:Cloudflare had a partial outage by Cthulhu_ on Tuesday June 21, 2022 @04:36AM (#31821690)
You say that, but there's tons of automated attempts doing the rounds on everything directly connected to the internet; centralized providers like Cloudflare can detect and prevent these patterns, whereas you need to be on the ball yourself if you have a service directly open to the internet. Exploits are exploited quickly, and while I make no assumptions about your particular website / application, a lot cannot push an update on short notice.
Re:Cloudflare had a partial outage by scrollaway on Tuesday June 21, 2022 @06:47AM (#31822417)
You would lose your wager.

DDoS attacks are extremely common at just about any scale, even if you only have a few thousand users.

Cloudflare going down like this? Actually first time I remember it happening. There’s been downtime before but nothing so major.

Re:Cloudflare had a partial outage by londons_explore on Tuesday June 21, 2022 @04:09AM (#31821509)
The origin IP's are already open to attack for most users.

It's trivial to scan the whole IPv4 internet to find out which IP you are hosting your site on.

Re:Cloudflare had a partial outage by renonce on Tuesday June 21, 2022 @04:19AM (#31821575)
You need to block 443 for any other IP than Cloudflare. The IP list can be found at https://www.cloudflare.com/ips/.
Re:Cloudflare had a partial outage by dingdongthe on Tuesday June 21, 2022 @04:25AM (#31821621)
and/or use Authenticated Origin Pulls with a TLS client certificate.
Re:Cloudflare had a partial outage by renonce on Monday June 27, 2022 @08:58AM (#31893936)
That would leak your IP nevertheless. People can figure out that you're serving a specific website by inspecting your certificate on handshake without actually connecting.