News for hackers. Stuff that matters.
Re:Cloudflare had a partial outage by jgrahamc on Tuesday June 21, 2022 @02:49AM (#31820777)
Yes, not worldwide but a lot of places. Problem with our backbone. We know what. Rollbacks etc. happening. Bring it back up in chunks.

Should be back up everywhere.

Re:Cloudflare had a partial outage by golanggeek on Tuesday June 21, 2022 @02:56AM (#31820880)
We are not using cloud flare. But our domain is also not accessible. We are using digital ocean's DNS service for propagating our IP. Does the DigitalOcean's DNS service depend on Cloudflare service?
Re:Cloudflare had a partial outage by tn890 on Tuesday June 21, 2022 @02:58AM (#31820918)
Re:Cloudflare had a partial outage by ThePowerOfFuet on Tuesday June 21, 2022 @03:09AM (#31821049)
That link isn't accessible from where I am right now.

Alanis Morissette agrees that this is ironic.

Re:Cloudflare had a partial outage by mproud on Tuesday June 21, 2022 @03:16AM (#31821112)
Yes, this is actual irony, unlike “rain on your wedding day.”
Re:Cloudflare had a partial outage by asddubs on Tuesday June 21, 2022 @06:10AM (#31822235)
but in regards to singing a whole song about things that are not irony being irony: isn't it ironic?

don't you think?

Re:Cloudflare had a partial outage by chrisweekly on Tuesday June 21, 2022 @06:57AM (#31822479)
Yes! IRL Alanis is smart and eloquent, and those that think that song's lyrics are evidence to the contrary are missing the joke.
Re:Cloudflare had a partial outage by baobob on Tuesday June 21, 2022 @08:14AM (#31822958)
Moved all my domains to DO specifically to stop donating traffic data to Cloudflare. Absolutely stunned I didn't notice this earlier.

It's bullshit all the way down.

Are there any companies left offering free DNS usable from Terraform that aren't part of the "Internet Five Eyes"?

edit: looks like Linode may be the next best 'not terrible' option

Re:Cloudflare had a partial outage by mnordhoff on Tuesday June 21, 2022 @01:55PM (#31827007)
I have bad news for you, Linode's authoritative DNS service also uses Cloudflare DNS Firewall.

  $ dig +short ns1.digitalocean.com aaaa
  2400:cb00:2049:1::adf5:3a33
  $ dig +short ns1.linode.com aaaa
  2400:cb00:2049:1::a29f:1a63
Re:Cloudflare had a partial outage by noobermin on Tuesday June 21, 2022 @03:01AM (#31820952)
So much for whoever downthread said "not all websites use cloud flare"
Re:Cloudflare had a partial outage by mercer on Tuesday June 21, 2022 @02:57AM (#31820909)
Same here, except DNSimple.
Re:Cloudflare had a partial outage by Deukhoofd on Tuesday June 21, 2022 @02:58AM (#31820914)
Yes.
Re:Cloudflare had a partial outage by mikesabbagh on Tuesday June 21, 2022 @04:26AM (#31821629)
CF SRE team need to rethink their published SLA of 100%. This is not reasonable. https://www.cloudflare.com/business-sla/
Re:Cloudflare had a partial outage by shric on Tuesday June 21, 2022 @05:58AM (#31822171)
An SLA of 100% simply means you agree to compensate your customers (as specified, usually with credit) if your service is down at all, nothing more.

Also, SRE here but not for Cloudflare -- I've never seen SREs directly involved in externally published SLAs, they usually come from legal. We deal with SLOs on more fine grained SLIs than overall uptime

Re:Cloudflare had a partial outage by codethief on Tuesday June 21, 2022 @06:46AM (#31822412)
> SLA […] SRE […] SLOs […] SLIs

I made it to SLA (which I believe stands for service level agreement). What do the other abbreviations stand for?

Re:Cloudflare had a partial outage by shric on Tuesday June 21, 2022 @06:52AM (#31822441)
SRE - Site Reliability Engineer (a term Google came up with that's been adopted elsewhere) Google defined it approximately as what happens when you apply software engineering practices to what was traditionally an operations function.

SLO - Service Level Objective - the service level you strive for. If it's higher you have room for experimentation, etc.

SLI - Service Level Indicator - the actual metric(s) you use to measure a service level (latency, error rate, throughput, etc.)

Re:Cloudflare had a partial outage by nsheridan on Tuesday June 21, 2022 @06:58AM (#31822489)
SLA - correct. That’s the contract between the operator and the users which describes the penalties for not meeting agreed-upon SLO

SLO - service level objective, the stated availability (or latency or durability etc) of the service. Usually expressed as a value over a period of time (e.g 99.9% availability as measured over a moving 30 average). The SLO is measured by the SLI.

SLI - service level indicator. Simply, the direct measurement of the service (i.e metrics)

SRE - Site Reliability Engineer, usually a member of a team who is responsible for the continued availability of the service and the poor sap who gets paged when it breaches SLO or has an outage or other impactful event.

Re:Cloudflare had a partial outage by KarlKode on Tuesday June 21, 2022 @07:15AM (#31822595)
SRE: Site reliability engineer

SLI: Service level indicator (Metric to measure the health of a service. For example successful requests per interval / total requests per interval.)

SLO: Service level objective (what performance you expect eg. the previously mentioned SLI is >= 99.5%)

SLA: Servicelevel agreement (legal agreement that defines what happens if a SLO is not met)

Re:Cloudflare had a partial outage by stn_za on Tuesday June 21, 2022 @04:42AM (#31821731)
Yep, I'd promise 99.95 at a stretch, never 100%.

They are not being honest with themselves here

Re:Cloudflare had a partial outage by latch on Tuesday June 21, 2022 @05:53AM (#31822157)
I'm not sure you and your parent understand what an SLA means. It's an agreement that, when broken, incurs a penalty.

They aren't saying they guarantee 100% uptime. They're saying they'll pay you for any downtime. It's literally the 3rd paragraph:

> 1.2 Penalties. If the Service fails to meet the above service level, the Customer will receive a credit equal to the result of the Service Credit calculation in Section 6 of this SLA.

(Most people I know consider them meaningless marketing BS that's really just meant to trick people or satisfy some make-work checkbox)

Re:Cloudflare had a partial outage by greyface- on Tuesday June 21, 2022 @06:04AM (#31822205)
> They aren't saying they guarantee 100% uptime

> Cloudflare ("Company") commits to provide a level of service for Business Customers demonstrating: [...] 100% Uptime. The Service will serve Customer Content 100% of the time without qualification.

This is a legal commitment to provide 100% uptime. They are guaranteeing 100% uptime and defining penalties for failing to meet that guarantee. The fact that a penalty is defined does not stop it from being a guarantee.

Re:Cloudflare had a partial outage by e1g on Tuesday June 21, 2022 @06:37AM (#31822362)
No, this SLA is a legal commitment to give you credits when Service uptime falls below a certain threshold. The threshold could be anything - 99%, 50%, 100%, etc. Importantly, Cloudflare is not under a legal obligation to provide the Service at or above the agreed threshold, it's under a legal obligation to give you Credits when the Service uptime is below that threshold.

"Service Credits are Customer’s sole and exclusive remedy for any violation of this SLA."

Re:Cloudflare had a partial outage by chrisseaton on Tuesday June 21, 2022 @06:53AM (#31822450)
> This is a legal commitment to provide 100% uptime. They are guaranteeing 100% uptime

I don't think you know what a guarantee is.

For example when you buy a new car you get a guarantee that it won't break down. Are they claiming it won't break down? No, of course not. What a guarantee means is that they'll fix it or compensate you if it does.

Re:Cloudflare had a partial outage by brokenkebab2 on Tuesday June 21, 2022 @06:30AM (#31822330)
Looks like it supports parent opinion: commit - bind to a certain course if policy. It's legal obligation, not a statement about guarantees in physical world (like "this alloy won't melt below t°C")
Re:Cloudflare had a partial outage by subvarad on Thursday June 23, 2022 @01:14PM (#31852182)
I completely can understand your emotion. But even the top CDNs can have outages of some form or the other. If site uptime is important, check out https://www.cdnreserve.com/ - it's built on the design principle that the likelihood of two separate platforms having an outage at the same time is close to zero.
Re:Cloudflare had a partial outage by arccy on Tuesday June 21, 2022 @04:42AM (#31821727)
that just means they're willing to pay for the marketing number, not that they will actually achieve it
Re:Cloudflare had a partial outage by samcheng on Tuesday June 21, 2022 @02:52AM (#31820812)
Thanks for being here with timely updates! I knew to come to Hacker News once the alert triggered and a few users started complaining.
Re:Cloudflare had a partial outage by fragmede on Tuesday June 21, 2022 @04:25AM (#31821625)
Good thing HN doesn’t use them then!
Re:Cloudflare had a partial outage by lesmond on Tuesday June 21, 2022 @02:56AM (#31820873)
Enterprise support has been useless as is the status page. Got more info here
Re:Cloudflare had a partial outage by dx034 on Tuesday June 21, 2022 @03:06AM (#31821017)
The status page shows about as much information as the post here.
Re:Cloudflare had a partial outage by lesmond on Tuesday June 21, 2022 @03:09AM (#31821044)
It didn’t at the time.

Their phone line kept cutting us off and then the people there were not too helpful.

Re:Cloudflare had a partial outage by robbiep on Tuesday June 21, 2022 @03:43AM (#31821331)
Status page was down for me in Sydney
Re:Cloudflare had a partial outage by Akuehne on Tuesday June 21, 2022 @02:57AM (#31820888)
Just wanted to also reiterate how thankful I am that you took your time to let us know. It speaks volumes.
Re:Cloudflare had a partial outage by taf2 on Tuesday June 21, 2022 @03:07AM (#31821022)
Agreed- couldn’t figure out what was going on… finally checked here and - ah now I can sleep
Re:Cloudflare had a partial outage by Abishek_Muthian on Tuesday June 21, 2022 @03:12AM (#31821073)
Cloudflare going down is one of the things which keeps me awake, My main complaint about Cloudflare is that they are very good at everything they offer that we've become reliant on them for everything.
Re:Cloudflare had a partial outage by brightball on Tuesday June 21, 2022 @06:26AM (#31822308)
Happens to everybody sometime. AWS seemed to have a major outage a couple of times a year for a while there.
Re:Cloudflare had a partial outage by subvarad on Thursday June 23, 2022 @01:06PM (#31852060)
Exactly but the likelihood of two networks going down at the same time is close to Zero. Check out: https://www.cdnreserve.com/ We rolled it out to complement top CDNs.
Re:Cloudflare had a partial outage by Abishek_Muthian on Tuesday June 21, 2022 @01:33PM (#31826754)
True, They're usually due to issues with BGP routes.

It's common to see CF being the DNS/CDN for applications across AWS, GCP, Azure etc. So perhaps CF being down affects more applications than individual cloud platforms?

Re:Cloudflare had a partial outage by jimmygrapes on Tuesday June 21, 2022 @03:33AM (#31821255)
"do no evil" springs to mind -- once burned, etc.
Re:Cloudflare had a partial outage by Abishek_Muthian on Tuesday June 21, 2022 @05:13AM (#31821917)
Yeah, What's up with the competition to Cloudflare? What's the real barrier for entry?

It's not infrastructure anymore, As there is a new PaaS startup every week offering distributed hosting and So why bundling in DNS, DDOS detection+mitigation, cloud workers... with it is so hard?

Re:Cloudflare had a partial outage by richdougherty on Tuesday June 21, 2022 @06:27AM (#31822310)
This is just my take, but Cloudflare looks to be building a "moat" to make entry hard. This is built around two things: 1. economies of scale, 2. a network effect.

-

https://en.wikipedia.org/wiki/Economies_of_scale

As Cloudflare gets bigger, they can provide services more cheaply. This is because (a) they can more fully utilise their data centres and other physical capital investments, (b) they can divide their fixed software costs over more users and (c) they get process efficiencies and discounts with scale.

A new entrant will struggle to match cost unless they're able to obtain similar scale. The bigger Cloudflare gets, the bigger the scale that a new entrant needs to hit before they can match them on cost.

-

https://en.wikipedia.org/wiki/Network_effect

Second they're aiming to build a network effect through having huge number of locations. The more locations, the more appealing to new customers as they can be close to more users. A competitor will have to build a similar number of locations to match Cloudflare's proposition.

A new entrant cannot provide as much value, and therefore cannot charge as high a price, without building a similar sized network. This again requires the entrant to invest heavily before they can charge a similar price.

-

The combination of these two things mean that when Cloudflare is operating at a large scale with a large network it can offer a more valuable service (and charge a higher price) than a new entrant, and earn more profit because it can operate at a lower cost.

Also, Cloudflare has the option of lowering its price and still being profitable due to lower costs at its scale, so it can deter entrants from trying to compete by the threat of being able to lower prices below what is profitable for new entrants.

The only players who can compete may be those who already have comparable size - Amazon, Google, Microsoft, Facebook, CDNs, etc, since they will already have addressed the issues of scale and network effects. However, they may not want to cannibalise their existing markets. It will be hard for other new entrants to compete.

Re:Cloudflare had a partial outage by subvarad on Thursday June 23, 2022 @01:07PM (#31852090)
There are many noteworthy players - Akamai, Fastly etc., and Edge plaoviders like ourselves (Zycada) who complement top CDNs like Akamai, Cloudflare, Fastly.
Re:Cloudflare had a partial outage by Abishek_Muthian on Thursday June 23, 2022 @10:54PM (#31857316)
The main difference between Cloudflare and the others mentioned is the price; One can start with CF for a side project for free and continue to use it free till it becomes a viable startup.

Others at best offer a limited trial plan, But most are just 'Speak to expert/ Contact us' for pricing which means haggling with a sales rep while we can just build things. Even the paid plans of CF is reasonable when compared with others with better features.

Re:Cloudflare had a partial outage by jgrahamc on Tuesday June 21, 2022 @05:31AM (#31822019)
It's hard at scale.
Re:Cloudflare had a partial outage by Abishek_Muthian on Tuesday June 21, 2022 @05:40AM (#31822078)
Isn't everything harder at scale? That's not a barrier for entry though.
Re:Cloudflare had a partial outage by Sohcahtoa82 on Tuesday June 21, 2022 @11:28AM (#31825041)
Building a CDN absolutely is hard to do at scale.

You can't build a Cloudflare competitor in AWS/Azure/Linode/DO/etc. You need your own data centers. Multiple of them across the country, ideally around the world if you want to serve the whole world.

This is insanely hard.

Re:Cloudflare had a partial outage by andrewnyr on Tuesday June 21, 2022 @10:09AM (#31824140)
for a global cdn... it quite literally is
Re:Cloudflare had a partial outage by Abishek_Muthian on Tuesday June 21, 2022 @01:26PM (#31826655)
Point taken, For a Global CDN - scale is the point of entry.
Re:Cloudflare had a partial outage by llausa on Tuesday June 21, 2022 @03:14AM (#31821096)
hats off to you sir - would not want to be in your shoes right now but thanks for the updates
Re:Cloudflare had a partial outage by jgrahamc on Tuesday June 21, 2022 @03:16AM (#31821117)
I don't have shoes on.
Re:Cloudflare had a partial outage by stavros on Tuesday June 21, 2022 @03:45AM (#31821348)
That's the point, two people can't fit in them.
Re:Cloudflare had a partial outage by tomglynch on Tuesday June 21, 2022 @03:32AM (#31821250)
In tumultuous times (fix wasn't implemented at this point), the Cloudflare CTO still has time for some wit. Love it!
Re:Cloudflare had a partial outage by keyle on Tuesday June 21, 2022 @03:27AM (#31821214)
It's back wooo!

And I'm saying this for the last time: no one type google into google!

Re:Cloudflare had a partial outage by bilekas on Tuesday June 21, 2022 @03:09AM (#31821043)
Thanks for the update, just curious if we will get a report on what happened ? In as much detail as can be of course - morbid curiosity mainly. I love the post reports these events usually bring.
Re:Cloudflare had a partial outage by blue_cookeh on Tuesday June 21, 2022 @03:13AM (#31821087)
Cloudflare are usually pretty good with posting post mortems. https://blog.cloudflare.com/tag/postmortem/
Re:Cloudflare had a partial outage by brycewray on Tuesday June 21, 2022 @08:55AM (#31823314)
Re:Cloudflare had a partial outage by halotrope on Tuesday June 21, 2022 @02:56AM (#31820875)
Thank you! This little comment just saved me an hour of investigation. Good luck for getting the system back up asap.
Re:Cloudflare had a partial outage by dangrossman on Tuesday June 21, 2022 @03:37AM (#31821290)
All my sites behind Cloudflare had come back up, and have now gone back to serving 500 errors.

The Cloudflare Dashboard is also no longer fully loading.

Re:Cloudflare had a partial outage by jgrahamc on Tuesday June 21, 2022 @03:41AM (#31821317)
Where are you located?
Re:Cloudflare had a partial outage by dangrossman on Tuesday June 21, 2022 @03:44AM (#31821334)
Raleigh, NC, USA

Sites are gradually reappearing as I type this. Some of my sites, and doordash.com, were returning 500 errors again just a minute ago. They just came back up, followed by the CF dashboard loading again.

Re:Cloudflare had a partial outage by hakanderyal on Tuesday June 21, 2022 @03:48AM (#31821371)
I'm from Turkey, and I also have been seeing intermittent errors for the last 10 minutes. Seems ok now.
Re:Cloudflare had a partial outage by janmo on Tuesday June 21, 2022 @03:02AM (#31820978)
Should probably do slower roulouts next time.
Re:Cloudflare had a partial outage by rcshubhadeep on Tuesday June 21, 2022 @03:09AM (#31821046)
Thanks for letting us know
Re:Cloudflare had a partial outage by b3lvedere on Tuesday June 21, 2022 @03:21AM (#31821169)
Thank you for your very fast support!
Re:Cloudflare had a partial outage by obnauticus on Tuesday June 21, 2022 @02:57AM (#31820897)
Which undersea cable was cut ;)
Re:Cloudflare had a partial outage by noitpmeder on Tuesday June 21, 2022 @02:58AM (#31820915)
Can't really roll back that change
Re:Cloudflare had a partial outage by ekimekim on Tuesday June 21, 2022 @02:59AM (#31820930)
Sure you can, but physically rolling back more cable takes a bit longer :P
Re:Cloudflare had a partial outage by MarkyMark210 on Tuesday June 21, 2022 @03:31AM (#31821238)
DR much ?
Re:Cloudflare had a partial outage by jgrahamc on Tuesday June 21, 2022 @03:44AM (#31821336)
I don't know what that means.
Re:Cloudflare had a partial outage by mike_d on Tuesday June 21, 2022 @03:56AM (#31821407)
DR means "disaster recovery," it is a formal plan used to respond to and mitigate potential risks to the business. Things like having a communications plan for an incident, or a backup office outside of your main office natural disaster zone.
Re:Cloudflare had a partial outage by jgrahamc on Tuesday June 21, 2022 @03:59AM (#31821426)
Ah. Just one more reason I hate acronyms. They obscure what the person is trying to say.