Thanks. Appreciate it. I’ll let the team look into it and communicate with you first.

I do wonder; when a domain is suspended, why don't you send out a courtesy notification e-mail?

what if their email address is @ the domain

Even if cloudflare stopped publishing the correct MX records for a domain, wouldn't Cloudflare still be able to send email to the MX server that had been specified? Seems like email should at worst be down for everyone but Cloudflare.

Well, that's poor planning on the user's part. In this case the OP stated he registered using a gmail address.

If the only contact information the registrar has is the @ of the domain then that likely mean that the registrar is failing the contractual obligations that exist between the registry and registrar. While I have not read the exact contract that exist for .com registrars, I am confident enough to say that you can not do that.

The OP made it clear that he used a gmail account for contact so there was no excuse for him to have not received a message regarding the deletion of his domain records.

I think it's pretty common advice that anything domain-related needs to be a different email. I've heard that, and I've only ever admin'd my personal domain. For exactly that reason I registered for Cloudflare with a gmail account.

Also display a prominent notification in the web interface then.

I appreciate you're looking in to this.

I’ve worked for Cloudflare since it was 24 people. I care a hell of a lot about our customers. I know eastdakota does also.

I can tell. Cloudflare has been a delight to use up to now and I'm grateful I'm getting this kind of support when I've only used your loss-leader services.

This is probably obvious but you're getting this kind of support because they're getting terrible PR by virtue of your post being at the top of HN. Not because he's such a great guy that cares so much about the customers. Not saying he isn't a great guy, or that anyone at cloudflare actually wanted this to be your experience.

If he’d emailed me directly I would have done the same thing I did. I emailed the head of engineering and support and asked for an explanation. I then jumped into the relevant chat room.

I do this sort of thing all the time. Sure, it’s unfortunate this is #1 on HN, but shrug. Fixing the problem and figured out what happened is important.

The more important question is what will be done to prevent this from happening again to somebody else (obviously this doesn’t need a response here, I assume an internal investigation is underway and this will be publicly communicated after its complete).

Hi John, your website's contact link is pointing to your twitter account and you don't have messages opened.

What is the best email to reach out to you?

Thank you.

How was he supposed to know to email you?

I don't think he's saying that OP should've known to email him, just that he would've done the same thing over private emails where nobody was watching, as he is doing here publicly.

To be fair this seems to happen with a bunch of tech companies, such as Stripe too. I wish more companies took customer support seriously.

I remember about 4-5 months ago I spent like 2 weeks going back and forth with Stripe's regular email support trying to understand their docs for SCA.

I kept getting a new rep who repeated the same things the previous reps were saying, which also had no bearing on what I was asking. It was basically a copy / paste from a script loop.

Then something negative about Stripe was on the HN front page and I happened to comment about a bad experience with the new SCA docs.

Within a few hours I was put in contact with a lead developer from Stripe who went as far as creating custom flow charts for my use case that wasn't covered in the docs and it was a pleasant experience, where "pleasant" felt like the person receiving the email was reading the words I wrote instead of just skimming them and pasting a boilerplate response.

But it only happened because of the HN comment. If that thread never appeared on HN, I'm not even sure I would still be using Stripe.

> was a pleasant experience, where "pleasant" felt like the person receiving the email was reading the words I wrote instead of just skimming them and pasting a boilerplate response.

Crazy how far down the bar has gone.

With many other companies I would agree with you, but in this case it is actually just because he's running a tight ship. I've had friends who've reached out directly via email and received the same level of support.

You mention loss-leader services, but the model is no doubt setup to funnel you towards profitable services.

I switched a personal domain to them, but it won't be long before I get my employer to move their resources over to them, assuming I see this all play out well.

John, you guys seem pretty awesome about posting root cause analysis and being transparent, can we expect to see at least a comment (if not a blog post) summarizing what took place here?

Has there been a followup update from Cloudflare?

Cloudflare followed up with me the day of and resolved the issue at least for me. I'm unsure if it would be appropriate for me to post the email they sent.

Maybe contact support first, figure out what happened and why before writing the blog post?

Maybe it's a cloudflare issue, maybe it's an honest mistake, maybe they are bad at customer service... either way, it's not great to shame a company before you even open a support ticket or talk to someone to find out.

I address this in TFA.

Essentially I felt that this was alright because when I filed a ticket I was informed that I should expect a long wait and that they recommend that their non-business customers post publicly on their support forum for crowdsourced support because that leads to faster replies. I was unable to log into that forum, and I suspect that may be because the way they set up SSO between the forum login and their main login may have failed in Firefox (with all tracking prevention and ad blockers disabled).

I felt that if a company invites me to ask for support publicly on their forum to save on customer support costs it's reasonable to talk about the issue in another public place.

Oh, the horror of recommending a public forum. (/s, of course.)

A fair number of questions aren't unique - product questions, how to use an API, etc. Someone may have asked a similar question, in which case you'll find an answer, find it faster than it'll take to hear back from the support team, and it deflects an unnecessary (already answered) ticket. That should be a win all around.

Now if you do have a novel question or something account-specific, by all means, open a ticket. There you'll get replies from people who can look up your account and give you specific answers.

The ombudsman tip in this post doesn't make a whole lot of sense when the normal support process wasn't really given a chance before making the blog post.

When your entire site and suite of applications are offline, especially during business hours, waiting who knows how long for a support queue isn't really an option. Outages like this can kill a business or cost millions.

(op here). To be fair to Cloudflare a business with millions on the line should be paying a lot more money to Cloudflare than I, a college student with significantly less on the line, would. If I had the option to pay $1,000 right now and get everything back up instantly I wouldn't take it because it isn't worth that much to me.

I realize that... the issue isn't necessarily tethered to your paying status, and it could just as easily happen to someone with a high value business account.

This is unlikely for a few reasons.

1. If you write into support as an Enterprise customer, you're basically getting an automatic response.

2. Enterprise customers can dial a direct line to support and have this under investigation within minutes.

3. Enterprise customer have a dedicated account manager, and in most cases a Solutions Engineer helped get everything setup. Usually Enterprise domains get locked, so they cannot get caught by linting type services. The likelihood of a domain being removed under those conditions is very low.

4. The most common cause of this type of thing is the Name Servers no longer pointing to Cloudflare, no one noticing for a while. Cloudflare periodically checks to see if a domain is still using CF name servers, and if they aren't they get moved as the assumption is they are no longer on Cloudflare. I don't know that this is what happened here, but it's easily the most common issue on lower tier plans. Enterprise companies often have people monitoring their infra, thus they catch this before Cloudflare conducts the removal of the domain.

Source: used to work on the Cloudflare support team.

> Oh, the horror of recommending a public forum. (/s, of course.)

He _tried_ to request help in their recommended public forum, but could not due to login issues.

Seems entirely reasonable to me to then go and ask about out in a different public forum?

And then/also submitted a support ticket, unless I'm misunderstanding their post?

> I spent some time thinking about if it was fair for me to post this on the same day as I filed a support ticket with Cloudflare.

I think it is a bit harsh to say that the OP was trying to shame anyone. He did post a support ticket. When your domain is "off-line" it is literally a shit hit the fan moment, and no one likes to be given the run-around.

To combat, hey my issue is always a Sev1 ticket, one can probably institute something like, here is a red button and if you click it, we will charge 100$. If it is indeed an issue that caused you to lose 90%(say) traffic and it was our fault, we will return the money.

Microsoft used to have this policy (maybe still does) for some support options. It cost $100 to open a case, and if the case was the result of a bug in the product, you got your money back.

Imagine how you would feel though if you were in OP’s shoes, where your DNS entries disappeared and client emails lost, through no fault of your own, but support won’t even listen to you until you pay money.

I think I'd be fine with that?

I would trust Cloudflare to pay me back, and I'm already putting something on the line. If it really was my fault this is going to be very embarrassing.

Edit: In case it isn't clear from context I'm the OP so I'm pretty sure I know what I feel.

Except you can bet that a lot of cases will end up with a battle of back and forth emails arguing over whose fault it was and who should pay the $100.

The vast majority of support tickets are the customers 'fault'. They'll be things like setting up DNS records incorrectly, believing they don't need to pay a renewal with their old registrar just because they moved their DNS over to cloudflare, frustrations over cached resources, etc.

(op here) My belief is that I completely transfered everything to cloudflare, paid in advance, and didn't touch it. If this was as much my fault as any of the common tickets you listed I will deeply regret having posted this and will apologise to Cloudflare.

In the blog post he mentions that Cloudflare recommends posting your problem to their open community forum to try and get things resolved. He couldn't access their forum, so he posted it on a different one.

And maybe not too.