I've put this idea forward a number of times here on HN in regards to other big tech companies.

Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.

The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company.

To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.

HN constantly has front page posts from people for whom big tech companies have support processes have failed but there is simply no other recourse unless you have "a friend in the business".

It just doesn't work to have some random Cloudflare person offer their email address as some post disaster issue resolution process on social media. Formalise it with an official Ombudsman and maybe then companies like Cloudflare might avoid HN front page bad publicity.

I had an issue at "one of the biggest tech companies" that went on for days and days in which tech support kept telling me I had set up something wrong, until eventually I emailed one of the top managers who I happen to "know" at that company - it was fixed within hours. That "contact a friend in the business who can actually get things done" is a necessary part of a large support organisation and it simply does not exist yet in any tech company that I know of.

> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.

What if the cost was put onto the business instead of the consumer and the business just hired support people who are all Ombudsmans by default?

Instead of focusing on copy / pasting boilerplate scripts and answering as many tickets as possible, they should focus on the problem the customer is having by default and do everything possible to reduce the number of incoming questions by fixing bugs, making a better product, improving their docs, etc..

I personally do around the clock email support for 35,000+ people who sign up to my programming courses and support isn't bogging me down. Relative to the number of minutes I'm awake, support is one of the least business related time consuming things I do per day, but I send individually personalized in-depth answers to everyone who asks me questions -- usually within an hour or less.

Because requiring the customer to pay a cost fixes the incentives problem. A customer may have an issue that is a minor inconvenience to them, that can either stay as not resolved, is due to error, or is simply not worth resolving to them. By having them pay for fixing the issue it could greatly reduce the number of inbound cases, allowing them to go straight to people that can act on them instead of sitting in triage and getting canned responses for O(days|forever).

I agree that in theory you can accomplish the same thing by making the product foolproof, but I don't think you can accomplish that for consumer facing products, and that doing so may not be a worth while trade-off. Additionally focusing on issues that greatly impact people rather than small things that cause friction with the product may (or may not! if it causes lack of retention) be worth more.

In terms of games theory; if there's no cost to the company to ignore the customer, then that's an incentive to ignore the consumer's complaint. Especially if that company has a monopoly.

I can understand why GP suggested company pays $20.

The suggestion is the customer pays $20 so they don't try and go straight to the ombudsman with ever little issue. The company doesn't and shouldn't pay the customer under any circumstance.

In technical support, one of the largest problems you have to deal with is all of the idiots that don't know how to use the product and won't look up the documentation or learn the product on their own. These people spam the shit out of support all day for the most basic shit. Seriously, I was working support at an anti-virus company and 85% of the calls into the paid business tier support were requests for us to do installs or basic application configuration for them. I know that not everybody knows how to configure a firewall, but "where do I download the install file for x" is literally Google-able.

The idea of an ombudsman is ultimately a bad one in this case because the problem isn't with the product or the support or the documentation. The problem is people Always think their issue is special and the most capable person should help them. Our support tier was paid, so more often than not we did what they asked, but the free customers would just get routed to a sales rep because free customers are even dumber and more entitled than the paying ones.

This whole situation is farcicle to me. Yet another free customer blew a minor issue out of proportion and had to apologize when it turned out the system hadn't failed him, he was just a free loading mooch all along.

A company has no incentive to ignore the customer, all else being equal, but they do have costs in servicing the customer and costs from ignoring the customer.

Assume the company has no way of distinguishing "valid" or "high impact" cases from other cases. This means that in order for the company to handle cases they need the sum of fully treating every case to be greater than the sum of the cost of fully treating every case. This is almost certainly never going to be true, unless each of your cases is high cost to ignore (think enterprise support).

So you need to funnel down the cases. Typically this is done with low tier support that tries to suggest fixes and such. You can also offer high value added support to give customers the ability to pay for a support plan. This $20 proposal is like that but on a more ad-hoc basis.

> if there's no cost to the company to ignore the customer

Simplifying, that's "if (false)", since ignoring the customers in the limit gets you a failed business and bankruptcy.

For the non-enterprise user, isn't this the scenario established today, with no failed business?

No, I don't think so. All the major internet companies listen to their non-enterprise users. They may not provide enterprise quality support to each individual user, but user voices in aggregate are considered critically important.

not if you have a monopoly. Which is why customer support for most ISPs in the US is so abysmal.

And even if it is just a sufficiently large company, if the customer is not a large payer and the issue is non-trivial to solve, the the cost of losing that customer could very well be less than the cost of fixing the issue.

What I was addressing was specifically the notion that companies pay zero cost for ignoring a customer, or the idea that they believe the same. That's false. Now there's no question there are some customers these large companies are better off without, and there are large and small examples of companies firing customers, or ignoring them til they go away. Nothing wrong with that. All the time on this site we talk about firing unprofitable customers. If you do this judiciously it's just good business. Doing that in particular cases is not the same as doing that generally.

NearlyFreeSpeech.net does something kind of like that. It costs money to raise any support ticket (you have to have a special support subscription) but you get refunded your "support points" if it turned out to be a problem on their end.

Because that isn't a useful allocation of resources.

Support that powerful are going to basically be devs. With dev salary expectations.

I think it's useful and it's very much worth it to give support a salary where they are enthusiastic to be employed.

If someone uses your product or service and has an issue, the first contact they have with your company is through support.

A crappy support experience could easily be the difference between having a life time customer valued at thousands of dollars, or your customer feeling frustrated and going to another company, netting you $0. That has a butterfly effect too because having 100 happy customers who are praising your support could lead to many new sales from organic recommendations. Having a bunch of customers who felt neglected by your support could yield a situation where your company is now on the front page of HN for having bad support or worse.

For whatever reasons, bigger companies focus more on measurable metrics like "tickets closed per hour", where the emphasis is on things that aren't important because measuring customer happiness is pretty subjective and doesn't translate well to employee evaluation scores.

I know if I ever got the point where I would need to delegate support assistance to someone, you can be sure I would pay them amazingly well, at least equal to a developer's salary because I never want anyone to ever feel like they get ignored or have a low quality experience with my products.

Airbnb is a great example of both. If you call tech support and get India, hang up. If you're lucky they're useless. If you're unlucky, they're going against your direct command and contacting the drug dealers in your house about a complaint. Contrast that to reaching one of the many on-shore agents who do their jobs well. One call, one or two people max and the issue is mostly resolved.

I've got no idea what the cause is. It's probably a mix of poor management, ineffective metrics, low salaries and a work culture in India which isn't synonymous with quality. The bottom line is, even the same company can be doing support wonderfully and terribly at the same time.

Yeah AirBnB support in India was awful and I wrote the company off after dealing with them for 20 minutes. Their onshore team actually cared about the huge cockroach infestation in our unit but the Indian guy I first talked to did not give a damn. And that’s who you’re most likely to get at 4 in the afternoon on your first day.

Part of "Wear the Customer's Shoes" at Twilio (when I was there) was that each quarter every developer and product manager was expected to spend a full day doing front line support for free(-ish) tier customers and however long it took to fully resolve the tickets they got during that time. Really let you understand the type of problems people had, how to have empathy, and basically how to make a better product.

I think this happens because it's very very difficult for a manager's manager to know if the frontline employee is spending their time solving real problems or slacking off with no actual regard for the customer.

"Metrics" are an unhappy middleground for everyone.

In my experience companies use not only "tickets closed per hour", but combine those with "average score" of 1-5 stars.

Yeah but at a large company the ratings don't matter too much in the end because it ends up being a spreadsheet game focused on hiring for the lowest cost while maintaining a level of support that's barely 1 notch above "horrible support".

So if you end up hiring someone who produces a bunch of 1-2 star ratings, you just fire them and try again with someone else until you get someone who produces mostly 3s+.

In the end you would think this would result in better support but it never does. You just end up turning over a lot of support employees, and the user experience for the customer is still bad because you need to first get through a robot menu, then talk to an entry level support who will happily let you pour your soul out on the details of the problem for 5 minutes, and then at the end they are like "oh, I can't do that, but I can forward you to someone who can".

And now you get to be put on hold again, anticipating by coincidence you'll probably lose the connection, and if you're lucky now you get someone who is capable of understanding the problem and you get to re-tell the whole situation again.

Before you know it, with wait times included you're 20 minutes into this and you just barely got to the point where you might get help for the issue. Businesses could solve this problem, but they don't. Instead of hiring better support folks for more money, they put the burden on the customer to have to wade through a bunch of BS and essentially train their entry level support for free -- and it's worse than free too because you're paying that company money for their service and you're trading time from your life to do it.

Yeah, even russian social network (200m users) have a bunch of very smart people supporting users from password recovery to psychological help. No way big startup can't afford something like this. You just have to do so.

I like the idea of the Ombudsman, but in this case I'm not sure that it's really applicable. I disagree that the "normal tech support processes have failed".

I wanted to point out something that may have been missed by many - which is that the OP in this case is a free user. i.e. he is NOT a "customer". He is a user of free services that Cloudflare is providing. It even says so in his post that he's a free account holder and that he's not entitled to tech support, hence why he kicked up a fuss online. My company is a Cloudflare Enterprise customer - when we write in, we get a response typically within an hour by their swat team.

The idea of an "Ombudsman" in this case wouldn't be an actual ombudsman in the spirit of advocating for the customer, but would be more of a "one time support fee" kind of deal. Which I think is fair, but in this case I'm not even sure the OP would have paid it. The OP had a viable option which would have been to upgrade to a Business account and get priority support.

I run a popular platform with 1.5 million registered users and the vast majority of them are free users. We also have the same problem. Most of our support is just swamped with free users mostly asking questions about how to do things (and it's in the documentation, they just don't read or search for it, it's too simple to just email support or blast away on Twitter and @ mention us). I've even had to withdraw from Twitter and Facebook entirely just because I was everyone's "ombudsman" (even if they weren't paying customers).

If you read his blog you would know that he is very much so a customer. To quote his blog verbatim "Because Cloudflare deleted my domain registration I can't change the status from clientTransferProhibited through their dashboard so I don't think I can even leave."

He registered his domain through Cloudflare Registrar, a paid service. I'd be concerned if my domain registrar told me to head to the forums for support for tld level issues that only the registrar can solve.

Ok but I still don’t get why the op complains about it so publicly the same day (likely immediately) after submitting the ticket without even giving the company a chance to triage and resolve the issue. (Yes he apologized) He paid $10 for a .org domain, which for Cloudflare is at cost. Goodness... It’s not like Cloudflare is refusing him support, just that he’s not on a tier where someone will respond in real time. In any case, it’s resolved for him now and he got the attention of the CTO (maybe I should try some public outrage sometime instead of paying for ent support heh!)

Also I think there’s an issue with what the expectations for support is for. The Cloudflare service itself - the DNS, DDOS protection, ssl, CDN, etc are all premium services with a free tier for kicking the tires. As far as I understood in the post, the OP was on that free tier but had registered the domain on Cloudflare (which was probably the added complication).

Sidebar - AWS offers no support even if you’re a paying customer - support is an additional product you have to add on and it is a percentage of your overall usage bill, even if you don’t contact support...

>Ok but I still don’t get why the op complains about it so publicly the same day (likely immediately) after submitting the ticket without even giving the company a chance to triage and resolve the issue.

Well the blog post isn't really primarily about his specific issue, it's about the systems being severely lacking. A ticket won't resolve that issue.

This is a really great idea, but I don't think it's possible for this to not get overused for every little issue. Once it's overused, it becomes useless.

Isn't this the point of the monetary hold? You can just raise it until the amount of entries become manageable. I'm sure higher level orgs would easily put down $10k to talk to a developer at Microsoft. For indie devs, numbers like $100 or $1000 could be manageable, as long as they can trigger the refund and close the case whenever they want.

Arguably this does block out poorer people from receiving "special" customer service, but there are not really other things people are willing to lose (or put up as collateral) for this type of service. I can't really ship Cloudflare my toaster or car until they resolve my case.

The first person to do that might at least be able to get their attention..

I like the setup my bank's Ombudsman has -- you must first take your issue to first level support, then escalate it with them if not resolved. If the second level of support denies you, then and only then can you reach out to the Ombudsman.

Any requests that haven't gone through the proper process get auto-rejected.

That is the logical way for things to work, but it requires first level tech support letting you escalate, which is not always the case with non-bank industries.

The current go-to move is to tweet a complaint at the company's Twitter account. This is surprisingly effective across multiple industries and actually was something my wife did that helped resolve a time-sensitive AirBnB issue.

This may be due to banks having regulatory requirements - they don’t want regulators in. Because of this, resolving issues before regulators are contacted is in everyone’s interest.

Tech companies being inundated with complaints would actually be expected. Making it easy to complain to ombudspersons will cause the complaints to dwindle naturally as tech companies stop the unethical, even criminal, behavior they engage in.

Very few people will complain about their lifetime Google ban after Google employs appropriate personnel to evaluate such cases.

Today very few people complain about Steam's refund policy, after Valve rewrote their refund policy to actually include refunds, after a judge ended their decade-long crime spree that saw an estimated 20,000 Australians robbed and an unknown quantity globally.

Here in Australia the ACCC (Australian Competition & Consumer Commission) has suggested tasking the TIO (Telecommunications Industry Ombudsman) with digital platforms.

Basically the way the TIO works like this: 1. The customer must first go through normal support until they get stuck. 2. The TIO sends the Company a nice letter requesting that they get their act together. 3. The Company then has 10 days to fix the issue or the TIO will escalate. 4. Either the problem got solved or the TIO keeps nagging until it is.

Every time the TIO has to mail the company it also charges the company for having to get involved. level 1: $31 level 2: $260 level 3: $475 level 4: $2250

If an issue gets to the end then the TIO can direct companies to implement solutions costing up to $50,000.

This process heavily incentivises companies to actually provide functional support. I don't think your idea of charging the customer $20 to get help with support would achieve anything.

I like the ombudsman idea. A lot.

I am a tech executive in my company and my direct e-mail address is at the bottom of every Web site. Yes, this means I deal with routing all sorts of problems BUT I know instantly when there is a problem. Any problem. And I can actually help. But really it helps me with my job. It's win/win/win.

Plus, poor customer services is simply inexcusable -- you have to treat people the way you want to be treated. You're letting yourself and everyone else down otherwise. There is ethics and morals in business and they are important.

It doesn't have to be my way but there are definitely ways to do it, do it well and not break the bank.

A properly setup helpdesk negates the need for an Ombudsman. If a tech company cannot get a helpdesk escalation path correct, what makes you think they'll get an Ombudsman scenario correct?

Ombudsman services are able to resolve conflicts where the customer and business don't agree. They're appropriate for regulated businesses to draw a line in the sand as to what is a legitimate complaint by independently settling cases where the customer and business did not agree. If the Ombudsman hears 100 unresolved complaints and upholds only one then probably the ordinary customer services are working as intended. If they uphold ninety nine something is gravely wrong with your customer services and the regulator needs to step in.

This isn't so dissimilar from the method used in pathology to deliver consistency in results across multiple labs and assay methods. You identify some boundary cases. This is definitely normal, this is definitely cancer but this is borderline, repeat a few times. You replicate and get all the labs to mark your samples. Then you can identify labs (or assay techniques) that aren't reliably putting things in the right categories and demand they improve or stop.

I've worked for various technical help desks over the past 20 years and I'm saying that if a help desk is setup properly, ombudsman services become unnecessary.

A big part of this is how empowered you make your various levels. The SLAs I've always worked with, created, and worked under measured time on calls and customer satisfaction was always one of the more important measurements.

You could also think of it as a "priority lane" for support. Pay nothing, get whoever for support. Pay $x, complaint goes to an engineer. Pay $y, the product manager. Etc. Skips the middleman process for customers that are in a rush.

If a customer is willing to put up 10k to get their issue resolved, it's probably an issue worth resolving.

Some companies, this doesn't work at all. EG: Postmates. Emailed pretty much every one of their executive/management team about a literally brand damaging issue, and received zero bounces, but also zero replies. Some companies have a policy to ignore unsolicited emails, no matter how serious the issue, as to not fuel the idea that doing so will get results in the future.

This is a really nice idea. This already exists for some companies in the form of Twitter accounts. I doubt the people doing the typee-typee actually has any authority over business processes, or demand a change, but I think they at least have the business owners on their speed dial, vs normal support tickets. But having an email is far better.

Amusingly, Twitter itself could use one. My account (@acangiano) has all of its images censored under "sensitive content" even though they are 100% benign images. No amount of tweets to @TwitterSupport has done anything at all to change it. There is basically no recourse. My account is like 12 years old and has 4.5K followers, so it's not like it's a random spam account, either. ¯\_(ツ)_/¯

I find this whole infrastructure, despite my taking advantage of it, to be very flawed. There're many customers who might have alarming issues, who never get attention because they're fearful to be perceived as a "Karen" for bitching on Twitter.

>they're fearful to be perceived as a "Karen" for bitching on Twitter.

On the other side of the equation sometimes I do want to complain for the sake of complaining without being harassed by some support account.

Especially hate getting obviously automated responses for daring to mention company names even if it would actually escalate to a human.

Yes it's a good point that companies currently have some sort of Twitter presence trying to address bad publicity posts, often they seem to be able to get things done.

The Ombudsman role is there to get things fixed when all else has failed, and before the angry customer posts to social media.

> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.

From what I read this is nothing to indicate the process failed (so far) just that the user decided to skip to the head of the line by writing a blog post internet style to get something resolved and attention. Failed is not 'I didn't get a reply or find what I needed as quickly as I think it should happen so now let me complain publicly so I get a reply'.

> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.

In theory nice but first it would be a 'deposit' and also opens up a host of new issues as far as the money being paid back and how that would be done and so on.

The core of what you say is correct. I posted about this publicly with two goals in mind: getting help from someone at cloudflare and getting advice on how to avoid this sort of issue from the HN community.

From my perspective Cloudflare's process did fail. Assuming I didn't do something insanely dumb and what I think happened did happen, I would consider that a failure on its own even if the support was perfect afterwards.

Copying from elsewhere:

I address this in TFA.

Essentially I felt that this was alright because when I filed a ticket I was informed that I should expect a long wait and that they recommend that their non-business customers post publicly on their support forum for crowdsourced support because that leads to faster replies. I was unable to log into that forum, and I suspect that may be because the way they set up SSO between the forum login and their main login may have failed in Firefox (with all tracking prevention and ad blockers disabled).

I felt that if a company invites me to ask for support publicly on their forum to save on customer support costs it's reasonable to talk about the issue in another public place.

This is a fair point. If users are directed to post publically for attention for support this user did the right thing.

Ive run into Cloudflare admin pages that fail with ad blocking before. Test your theory in a private session, it works for me when their site "has issues."

(I think in my case it was adding google metrics from the apps page.)

I would absolutely pay for this as a service. I just ran into an issue with AWS where there was a problem with my account, I was unable to send email from a specific IP in a pool of IP's bound to an ec2 instance.

The machine / IP combination has been working for years and then one day it just stopped. No emails from ec2-abuse, nothing in my support dashboard. After troubleshooting for a while I bit the bullet and ordered business support.

After cycling through three support agents over 48 hours Amazon said that they were filtering the IP because of spam complaints but neither the Abuse department nor my support liaison was able to see the block - it has been applied at the network level and the network team did not communicate the changes to the other departments.

I told them that I would not be paying for a support plan (over $1000) to resolve a problem that I did not have any notifications or alerts or any way to troubleshoot for myself.

It required talking to two manager and almost a month before they finally agreed that the billing should be refunded. After lots of support tickets, emails, and phone calls I got the money back - but man... An ombudsman would have been amazing.

I like this idea a lot. Some people below are suggesting that the ombudsman should be the last stop in a support queue if your problem isn't resolved, and that makes sense sometimes, but other times you can't wait that long!

So, all support systems should have a triage type system with a "nurse" having a constant eye on every new case that comes into the support system. When there's an emergency, such as the one associated with this post, then it should be forwarded to the ombudsman or some other emergency team immediately.

I think you're incorrectly assuming most companies want to provide good customer support.

That often ends up being the defacto job of a online community manager, if they have one.

> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.

All companies should have that!

This sounds nice in theory, but you know that tons of people would just go straight to the ombudsman, thinking they can jump the support queue or bypass established process.

The "shit filtering" workload would be tremendous..

The way this works everywhere is you go through the normal support process until you reach top level support if the situation isn't resolve you go to the ombudmen. If you go before you tried support they will direct you to support first.

Yes I understand the process, but as I said that "they will direct you to support first" work would be non-stop..

Not saying it doesn't work, just that a lot of people wouldn't follow the process, and anyone who sets this up should be prepared for a lot of triage work.

In Amazon's case, Bezos' email is just that. For AWS, Andy Jassy's (CEO) or Charlie Bell's (SVP) might do just as fine.

Source: Someone who emails them from time to time to get impasses resolved.

Just had to chime in here, but fifteen+ years ago, Bezo's email was ombudsman like, with him or a top level person reading it.

Now, it's just another level of very poor, very scripted support.

I'm fairly sure Amazon has taken the (perhaps wise, in a business sense) approach of not caring if a small percentage of users leave, due to support issues.

The cost of keeping customers with certain support issues, greatly outweighing supporting them.

This is why you have to hunt madly around Amazon's webpage to find contact info, why all forms of help point away from contacting a person, including their chat being bots now, until you move outside of their scripts.

Just one data point but, sending Bezos a nice hello and airing my grievances has worked for me every single time.

Obviously, Bezos may not read those emails but his aides and assistants who do have access to his inbox and act on the emails on his behalf do inherit his complete authority.

Some refs:

https://news.ycombinator.com/item?id=16341154

https://news.ycombinator.com/item?id=17193363

https://news.ycombinator.com/item?id=22286350

https://news.ycombinator.com/item?id=9356182

https://news.ycombinator.com/item?id=20782392

https://news.ycombinator.com/item?id=13512106

The problem is that big companies don't care about giving quality support for their products, and for the most part they get away with it. From their perspective there's no problem to solve.

Your solution basically boils down to "companies are failing at escalating support issues well, so they should escalate support issues well."

jgc is not a "random" Cloudflare person ;)

My point is precisely that to me, "jgc" IS a random person. How the heck do I know who this person is.

It shouldn't matter, and it should not be required, that someone "known and important" within an organisation decides to start doing hands on tech support in social media following a PR disaster.

If "jgc" is actually someone important within this company then maybe after fixing this issue, they can then go fiox their tech support by setting up and ombudsman and get their PR disasters off the front page of HN.

It’s the CTO according to their HN profile ;)

You're missing the point. It shouldn't be necessary for someone to know who's-who in order to get things resolved.

You may be right, but knowing "who's who" is very largely how general business gets done. Buying services over the internet from an anonymous black box with no support is a recent disruption.

No, normally you didn’t have to know someone in the C suite to “get business done”. That’s totally unscalable.

What’s a recent development is the complete lack of support when shit goes south. Back when you were interacting with real reps you had people that could see when stuff was obviously wrong and escalate appropriately.

Did I say anything about C-suite? Your point is nearly word for word the same as mine, I'm not sure why you're replying as a refutation.

For a regular generic Cloudflare customer like me, for personal use, jgc is one of the random Cloudflare person. I have started a spreadsheet with his name, email, comment link, and my copy of screenshot of comment; just in case if I need to email him anything in future.

[email protected] worked amazingly for me back when I bought apple products. Both times I literally get a phone call from someone who has already talked to my local store and has his proposed solution set up for me.

But that was like 10 years ago. Not sure if it's still good.

I can confirm, in those rare instances where it’s required, similar email addresses still work. Just keep it nice and polite. I’m sure they have manually maintained spam filters...

This is also the primary original source of Jeff Bezos’ legendary “question mark” emails.

> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.

Pretty much every technology company already have "ombudsman" for their important clients.

> That "contact a friend in the business who can actually get things done" is a necessary part of a large support organisation and it simply does not exist yet in any tech company that I know of.

I can assure you that every tech company has these "friends" available for their most important clients/customers.

Whether it makes sense for companies to make these contacts available to every customer is another matter.

> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.

At least in the case of telecom companies they all seem to have executive response teams when the usual channels fail. Send an email to any of their Exec team members (CEO, president, CTO, etc) and it nearly always gets assigned to a special team that solves the problem. I can confirm this out of personal experience with at least AT&T, Comcast, and T-Mobile.

Great idea.

Maybe it could be something that is given to someone when their ticket is closed (or maybe after the first tech response... it depends on the company/ corporate structure).

That way the ombudsman has something to work with, and would slow down the barrage that would occur by having a such a public contact point.

I'm never a fan of 'pay then get refunded' for something that's not your fault, and is entirely out of your control.

That's actually a really good idea. Bit like a Credit Card Chargeback, except it's charged to the customer and is refundable.

At 20 USD a pop, Google would make a small fortune!

I'd be the first to pay to get them explain to me some of their misterious weirdnesses.

$20 won't stop anyone with using it.

If it is refundable, make it $300. That is high enough that only people and businesses with a showstopper situation will use it.

$300 becomes a pretty high barrier for a lot of people. There a places with decent connectivity where that is a good weeks wages

I was talking within the US economy context. Prices obviously can differ globally.

But you get it back, the point is to make it a penalty if you are wasting their time.

> But you get it back

You're missing the part where some people, that actually would need this support, literally wouldn't be able to find that money because of the difference in purchasing power of the local currency.

I was thinking price for business not personal accounts. Maybe youre thinking those are the same thing in freelance.

No, He's thinking about various developing nations that are trying to have a tech boom.

Phillipines is a good example; Average yearly salary is somewhere around 12k USD, Median is 8k USD.

That means if someone is doing a tech startup there, 300$ is somewhere between a quarter and a half of some employee's pay for a -month-. And given the potential for tight margin/cash flow of startups even in the US... it would still price a lot of smaller players out of the market.

Prices can be different for different countries/customers.

A single dollar is a high barrier for some people.

You need an adjustable amount that is based on the annual income/revenue of the person/entity making the request.

Make it high enough to be non-trivial, but not so high that it blocks all effective usage of the safety valve.

Now, if you can solve that problem, I've got some bridges for you in Arizona.

I think the only sane thing you can do is price it at the cost it takes to review it. It will still be out of reach of some people, but at least its not arbitory.

Make it $20 but not refundable. Or $300 and not refundable.

This is a good way to filter the full level of importance. Most places that need a problem solved, now, dammit will be willing to pay.

And, if you want to be nice, you can refund it. Or, if the person was a jerk, you can keep it.

If I have a rightful complaint and it costs me $300 to resolve I am done with that business.

> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.

So basically you give the ombudsman $20, and they keep the $20 if they fail to resolve the problem?

> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.

HN actually acts somewhat like a crowd-sourced ombudsman. People who have an issue write a description and post it to HN. If enough people find it compelling, it makes the front page. Once it makes the front page, someone in authority at the involved tech company will see it, and try their best to resolve it.

So maybe HN should be billing them.

OP here.

You can post updates with any relevant information. Probably goes without saying, but if the issue has to do with my billing or address please don't post specific details without asking me first.

I will link to this comment from TFA for verification. (Edit: added to the bottom. If you need more verification you have my email.)

Edit2: I see that the domain is back in my account and listed as "Pending Nameserver Update". I don't think that's because of something I did.

Thanks. Appreciate it. I’ll let the team look into it and communicate with you first.

I do wonder; when a domain is suspended, why don't you send out a courtesy notification e-mail?

what if their email address is @ the domain

Even if cloudflare stopped publishing the correct MX records for a domain, wouldn't Cloudflare still be able to send email to the MX server that had been specified? Seems like email should at worst be down for everyone but Cloudflare.

Well, that's poor planning on the user's part. In this case the OP stated he registered using a gmail address.

If the only contact information the registrar has is the @ of the domain then that likely mean that the registrar is failing the contractual obligations that exist between the registry and registrar. While I have not read the exact contract that exist for .com registrars, I am confident enough to say that you can not do that.

The OP made it clear that he used a gmail account for contact so there was no excuse for him to have not received a message regarding the deletion of his domain records.

I think it's pretty common advice that anything domain-related needs to be a different email. I've heard that, and I've only ever admin'd my personal domain. For exactly that reason I registered for Cloudflare with a gmail account.

Also display a prominent notification in the web interface then.

I appreciate you're looking in to this.

I’ve worked for Cloudflare since it was 24 people. I care a hell of a lot about our customers. I know eastdakota does also.

I can tell. Cloudflare has been a delight to use up to now and I'm grateful I'm getting this kind of support when I've only used your loss-leader services.

This is probably obvious but you're getting this kind of support because they're getting terrible PR by virtue of your post being at the top of HN. Not because he's such a great guy that cares so much about the customers. Not saying he isn't a great guy, or that anyone at cloudflare actually wanted this to be your experience.

If he’d emailed me directly I would have done the same thing I did. I emailed the head of engineering and support and asked for an explanation. I then jumped into the relevant chat room.

I do this sort of thing all the time. Sure, it’s unfortunate this is #1 on HN, but shrug. Fixing the problem and figured out what happened is important.

The more important question is what will be done to prevent this from happening again to somebody else (obviously this doesn’t need a response here, I assume an internal investigation is underway and this will be publicly communicated after its complete).

Hi John, your website's contact link is pointing to your twitter account and you don't have messages opened.

What is the best email to reach out to you?

Thank you.

How was he supposed to know to email you?

I don't think he's saying that OP should've known to email him, just that he would've done the same thing over private emails where nobody was watching, as he is doing here publicly.

To be fair this seems to happen with a bunch of tech companies, such as Stripe too. I wish more companies took customer support seriously.

I remember about 4-5 months ago I spent like 2 weeks going back and forth with Stripe's regular email support trying to understand their docs for SCA.

I kept getting a new rep who repeated the same things the previous reps were saying, which also had no bearing on what I was asking. It was basically a copy / paste from a script loop.

Then something negative about Stripe was on the HN front page and I happened to comment about a bad experience with the new SCA docs.

Within a few hours I was put in contact with a lead developer from Stripe who went as far as creating custom flow charts for my use case that wasn't covered in the docs and it was a pleasant experience, where "pleasant" felt like the person receiving the email was reading the words I wrote instead of just skimming them and pasting a boilerplate response.

But it only happened because of the HN comment. If that thread never appeared on HN, I'm not even sure I would still be using Stripe.

> was a pleasant experience, where "pleasant" felt like the person receiving the email was reading the words I wrote instead of just skimming them and pasting a boilerplate response.

Crazy how far down the bar has gone.

With many other companies I would agree with you, but in this case it is actually just because he's running a tight ship. I've had friends who've reached out directly via email and received the same level of support.

You mention loss-leader services, but the model is no doubt setup to funnel you towards profitable services.

I switched a personal domain to them, but it won't be long before I get my employer to move their resources over to them, assuming I see this all play out well.

John, you guys seem pretty awesome about posting root cause analysis and being transparent, can we expect to see at least a comment (if not a blog post) summarizing what took place here?

Has there been a followup update from Cloudflare?

Cloudflare followed up with me the day of and resolved the issue at least for me. I'm unsure if it would be appropriate for me to post the email they sent.

Maybe contact support first, figure out what happened and why before writing the blog post?

Maybe it's a cloudflare issue, maybe it's an honest mistake, maybe they are bad at customer service... either way, it's not great to shame a company before you even open a support ticket or talk to someone to find out.

I address this in TFA.

Essentially I felt that this was alright because when I filed a ticket I was informed that I should expect a long wait and that they recommend that their non-business customers post publicly on their support forum for crowdsourced support because that leads to faster replies. I was unable to log into that forum, and I suspect that may be because the way they set up SSO between the forum login and their main login may have failed in Firefox (with all tracking prevention and ad blockers disabled).

I felt that if a company invites me to ask for support publicly on their forum to save on customer support costs it's reasonable to talk about the issue in another public place.

Oh, the horror of recommending a public forum. (/s, of course.)

A fair number of questions aren't unique - product questions, how to use an API, etc. Someone may have asked a similar question, in which case you'll find an answer, find it faster than it'll take to hear back from the support team, and it deflects an unnecessary (already answered) ticket. That should be a win all around.

Now if you do have a novel question or something account-specific, by all means, open a ticket. There you'll get replies from people who can look up your account and give you specific answers.

The ombudsman tip in this post doesn't make a whole lot of sense when the normal support process wasn't really given a chance before making the blog post.

When your entire site and suite of applications are offline, especially during business hours, waiting who knows how long for a support queue isn't really an option. Outages like this can kill a business or cost millions.

(op here). To be fair to Cloudflare a business with millions on the line should be paying a lot more money to Cloudflare than I, a college student with significantly less on the line, would. If I had the option to pay $1,000 right now and get everything back up instantly I wouldn't take it because it isn't worth that much to me.

I realize that... the issue isn't necessarily tethered to your paying status, and it could just as easily happen to someone with a high value business account.

This is unlikely for a few reasons.

1. If you write into support as an Enterprise customer, you're basically getting an automatic response.

2. Enterprise customers can dial a direct line to support and have this under investigation within minutes.

3. Enterprise customer have a dedicated account manager, and in most cases a Solutions Engineer helped get everything setup. Usually Enterprise domains get locked, so they cannot get caught by linting type services. The likelihood of a domain being removed under those conditions is very low.

4. The most common cause of this type of thing is the Name Servers no longer pointing to Cloudflare, no one noticing for a while. Cloudflare periodically checks to see if a domain is still using CF name servers, and if they aren't they get moved as the assumption is they are no longer on Cloudflare. I don't know that this is what happened here, but it's easily the most common issue on lower tier plans. Enterprise companies often have people monitoring their infra, thus they catch this before Cloudflare conducts the removal of the domain.

Source: used to work on the Cloudflare support team.

> Oh, the horror of recommending a public forum. (/s, of course.)

He _tried_ to request help in their recommended public forum, but could not due to login issues.

Seems entirely reasonable to me to then go and ask about out in a different public forum?

And then/also submitted a support ticket, unless I'm misunderstanding their post?

> I spent some time thinking about if it was fair for me to post this on the same day as I filed a support ticket with Cloudflare.

I think it is a bit harsh to say that the OP was trying to shame anyone. He did post a support ticket. When your domain is "off-line" it is literally a shit hit the fan moment, and no one likes to be given the run-around.

To combat, hey my issue is always a Sev1 ticket, one can probably institute something like, here is a red button and if you click it, we will charge 100$. If it is indeed an issue that caused you to lose 90%(say) traffic and it was our fault, we will return the money.

Microsoft used to have this policy (maybe still does) for some support options. It cost $100 to open a case, and if the case was the result of a bug in the product, you got your money back.

Imagine how you would feel though if you were in OP’s shoes, where your DNS entries disappeared and client emails lost, through no fault of your own, but support won’t even listen to you until you pay money.

I think I'd be fine with that?

I would trust Cloudflare to pay me back, and I'm already putting something on the line. If it really was my fault this is going to be very embarrassing.

Edit: In case it isn't clear from context I'm the OP so I'm pretty sure I know what I feel.

Except you can bet that a lot of cases will end up with a battle of back and forth emails arguing over whose fault it was and who should pay the $100.

The vast majority of support tickets are the customers 'fault'. They'll be things like setting up DNS records incorrectly, believing they don't need to pay a renewal with their old registrar just because they moved their DNS over to cloudflare, frustrations over cached resources, etc.

(op here) My belief is that I completely transfered everything to cloudflare, paid in advance, and didn't touch it. If this was as much my fault as any of the common tickets you listed I will deeply regret having posted this and will apologise to Cloudflare.

In the blog post he mentions that Cloudflare recommends posting your problem to their open community forum to try and get things resolved. He couldn't access their forum, so he posted it on a different one.

And maybe not too.

Please explain something to me.

For me and my of my customers, having your "entire cloud deleted" is like... the #1 nightmare scenario.

So why does this capability/function even exist for active accounts at CloudFlare? It sounds like the OP fell victim to what is essentially a regular process.

Or to put it another way: No amount of explanation or assurance is ever going to make me feel comfortable with my doctor having a handgun as one of his medical instruments.

It doesn't sound like it actually deleted anything other than the DNS entries. It's not like any actual data got lost.

I dunno, but DNS entries are actual data.

I certainly don’t know what all the DNs entries on my 15 domains are, so if they disappear it’s not going to be easy to get it all back.

> I certainly don’t know what all the DNs entries on my 15 domains are, so if they disappear it’s not going to be easy to get it all back.

I hope for your sake that right now you are fixing that problem?

More like another thing has been added to the TODO list.

(OP here)

It appeared to me as though they deleted my domain registration from their system. That's mostly why I freaked out. I wasn't sure if I'd even be able to transfer my registration easily.

However, on the advice of someone from HN, I found that if I went through the flow of linking an external domain to my cloudflare account the registration details came back.

Why can you delete DNS records? Well it's pretty common to want to do so (usually followed by adding some new ones).

Your doctor has a syringe as a medical instrument. They can kill if not used correctly.

This is not the user manually deleting records.

This is the doctor seeing a patient that hasn't moved in a few hours and executing them:

https://community.cloudflare.com/t/does-cloudflare-automatic...

Oh wait, he was just sleeping? Ah, well... "oops".

There is exactly one reason you'd delete customer data without their active involvement: They're overdue on their bill and have ignored all warnings and communications. That's it. There are no other reasons.

Feel free to disable things if they're a security risk, but Do. Not. Delete. Customer. Data.

Ever.

Ugh! I may have interpreted the comment I responded to wrong, and replied to my own strawman. I think you need the power for the user to delete records (obviously) but this could mean a bug in the system could cause the records to be deleted. But it sounds from what you are saying that they sort of "intentionally" deleted the records (not sure if a human had the intent or an algorithm?) but if intentionally deleted yes that sucks, there is no reason for that I can think of, given what they do is about handling DDOS.

Unless the mechanism exists for legal reasons, i.e. they are forced to take some sites down by the government or law suits.

Having the CTO offer up support in this way is heartening. Especially because infrastructure suppliers are getting more and more centralized, which leads to possible SPOFs even with individual company distributed environments.

The problem is that big companies don't care about giving quality support for their products, and for the most part they get away with it. From their perspective there's no problem to solve.

Your solution basically boils down to "companies are failing at escalated support issues well, so they should escalate support issues well."

I posted this under the wrong parent comment, moved but cant delete.

Context for the lazy, jgrahamc is the longtime CTO of Cloudflare.

Please do update if possible.

It's likely a good learning for all.

Of course, I just don’t want to promise something when it might be revealing information about an individual account.

He gave me an incredibly detailed and fast response to an issue I was fact-checking.

First, you are awesome, really :)

Second, a bunch of honest questions:

Did you consult to your supervisor (or anyone with authority) to be able to bypass the support process (if there is any) like this? If so what was the response? If the response was negative, how did you convince people? After things resolve, can you kindly post how many spam or unrelated emails you receive so that it will be an example to the industry?

I'd like to put my skepticism on hold and blindly believe that your post is a reflection of pure concern and not just a PR stunt for damage control.

The CTO at Cloudflare has to ask his supervisor before trying to help someone over at hacker news? :)

I didn't know which probably slightly proves some point other people made.

jgrahamc is the CTO of Cloudflare. His only supervisor is eastdakota/ Matthew Prince, the CEO of Cloudflare.

Then I'd like to know the CEO's opinion then.

CTO of any company doesn't need the CEO's permission to make a customer relations post.

It's not about permission but cooperation of two disciplines.

It's hard to understand criticism when it is not put into words.

From my perspective, a developer from the company the article is about arrives in the comment section. I ask an honest question about the communication process internal to the company. I am told this developer is the CTO and is supervised by the CEO. Then I state the fact that I would appreciate if we also knew what the CEO thinks about this situation.

I don't literally expect answers from the people I addressed and that's OK, they are probably too busy working on the subject matter. If they take time and write one, that is really admirable and I'd be thankful.

What I don't understand is why it is considered inappropriate to be curious about these topics, as judged by the amount of downvotes I received.

Maybe the tone I intended for my post got lost in translation, since I'm not a native English speaker.

It comes across as entitled, and also just strange and unnecessary. Like, why would you need or expect to hear directly from the CEO of Cloudflare when we're already hearing from the CTO? I'm not sure why your reply here got downvoted as well though, as I found it helpful that you explained you were simply interested in the CEO's position rather than actually expecting a response from him. So, just trying to illustrate how the initial comment came across to me, and perhaps others.

Thanks for the feedback.

"I'd like to hear CEO's opinion" probably doesn't mean "I wonder what the CEO thinks about public communication by his/her colleagues inside internet forums". I'm reminded English is hard even when I think I have a basic understanding of common speech patterns (parlance?).

Correct. In fact, even the second phrasing, "I wonder what the CEO thinks..." could easily be interpreted as expecting a response. It seems you'd have to really bend over backwards to avoid that, perhaps explicitly saying something like, "Obviously I wouldn't expect him to respond here, but I'm curious whether the CEO shares this view." Or something like that.

This reply might be what you're looking for: https://news.ycombinator.com/item?id=22407399

Cloudflare is a solid company with a good reputation and has done a lot for general welfare of the internet. The staff there generally care about customers and it seems like they're trying to figure out what went wrong here, and of course preventing negative PR is always a bonus.

There's really not much of a process when you're a top officer at a company, you have the freedom and responsibility to exercise your judgement and do whatever it takes for the good of the organization.

> BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is [email protected] and I’m happy to hear from people.

I know that people will think it's great that you are doing this and I also know that you think it's good (for you) to have a feel for the issues that frustrate every day users. But I think it's not a great use of a company execs time and I am not even sure it's a good way to deploy resources at Cloudflare.

The reason is people will tend to (as a rule) do as little as they can themselves but then use as a hammer the court of public opinion to get something resolved.

You say 'ever find yourself so frustrated with Cloudflare' but you know that in itself is different for different people. What will happen is you will get people using you as a help desk and then after you don't help them as quickly as they think you should they will then follow up with a post, comment or story about how you did nothing.

Separately if someone is posting publicly about an issue (as this person is) and if you can verify that it's actually coming from the customer (I mean who says it is actually?) I don't think you need them to say it's ok to resolve online. In fact to me it's the opposite. You take the time to reach out publicly and you take what follows good or bad even calling you out (the customer yes you can do that by the way) if you think they didn't put the appropriate effort into finding an answer.

I disagree with you. It matters enormously that people like me are available. Sure there are time wasters who’ll send me email. But I don’t care. Dealing with a small number of real customers doesn’t take a lot of my time and matters.

Everyone optimizes for the worst case. They think “if I give out my email I’ll get tons of useless email”. I can assure you I get 10x the crap sent to me on LinkedIn than via direct emails from customers or others.

Thank you, this is a good attitude to have and I wish more company exec’s thought like this. We might have fewer post’s like OP’s story.

My little firm is nothing like the scale of CF but I do the same and have dropped my email address around a fair bit. I do get some pretty interesting missives that make it past the filters but signal to noise is very high in my inbox.

I deleted my Linkedin account well over a year ago and I still get emails from them saying my profile is being viewed. Tossers.

If I ever have someones direct email I don’t think I would use it in any situation that I wouldn’t appreciate getting one myself.

> I know that people will think it's great that you are doing this and I also know that you think it's good (for you) to have a feel for the issues that frustrate every day users. But I think it's not a great use of a company execs time and I am not even sure it's a good way to deploy resources at Cloudflare.

As a counter example, Jeff Bezos (whose time may be worth more than anybody else's) famously audits his email for customer complaints and occasionally derails an organization for a day or two in order to figure out what happened. He stands behind this practice and has said that he often picks out cases where the anecdotal complaint is counter to data that he's been presented, and that more often than not the anecdotes are correct and find a shortcoming in the data. IMO it also demonstrates a culture of caring and following up about anecdotes to others whose time is worth less than his own.

> As a counter example, Jeff Bezos (whose time may be worth more than anybody else's) famously audits his email for customer complaints and occasionally derails an organization for a day or two in order to figure out what happened.

Is that why Amazon is growing more and more notorious for selling fraudulent items over the years?

Both of those things could be true without one being causal or even related.

Maybe someone needs to email Jeff about that.

Now that the OP has given a go ahead to go public, We will eagerly wait for your update jgc

Not a fanboy, still downvoted your post because it's incredibly whiny and does not contribute anything. All you're doing is accuse of censorship without evidence, you say they screwed up "so many times" yet fail to show one example, and then you preemptively accuse people who would dare downvote you to be fanboys.

Sorry, but that comment is noise. Mine is too, but hopefully it helps you see things more clearly rather than let you pat yourself on the back thinking "HN is full of fanboys anyway".

When you use the phrase "un-American" please remember that you're talking to an international audience who see being American as a completely orthogonal issue to any type of behaviour. It makes you seem parochial, and slightly nationalistic.

To be clear, the parent comment is written by someone who is upset that CloudFlare terminated service to violent neo-Nazis. The fact their language is nationalistic is unsurprising.

Violent? I'm not aware of any lawsuits due to violence by the neo Nazi founder and I despise neo Nazi speech as anyone else.

You completely missed the point or maybe you just don't give a s++t about 1A rights.

The other issue is they claimed to be hands off to customers, obviously within the bounds of the law.

The sad truth is this - distasteful as DS/Anglin is, he was not running afoul of the law. Matthew Prince unilaterally decided to reverse 200+ years of 1A jurisprudence for one of its customers because he disagreed with their views.

These Matthew Prince types are the same assclowns who sue Christian bakeries for refusing to adorn wedding cakes with 2 male or 2 female figures when they exercise their 1A rights to discriminate based in faith preferences.

The height of hypocrisy displayed by Cloudflare was galaxy sized, despite it being legal.

That's what I and many others blasted them about.

I can't put it any more clearly than this.

> "the same assclowns who sue Christian bakeries for refusing to adorn wedding cakes with 2 male or 2 female figures when they exercise their 1A rights to discriminate based in faith preferences."

How is this different from refusing service to neo-Nazis?

It seems like you're arguing that the first amendment should apply if I refuse to sell a cake to a gay couple, but not if I refuse to sell DNS service to neo-Nazis.

Can you clarify?

You only get sued if you're defending your faith. That's the difference.

Your argument is not philosophically consistent, and it's clear you are seeing persecution rather than philosophy. Under Conservative philosophy, it is clear that CloudFlare has a right to deny service to a customer it doesn't want. It's chosen to not provide services to violent neo-Nazis. Get over it.

Please explain to me how it's un-American for a company to choose who they want as their customers? The typical Conservative position would be that businesses have a right to choose who they wish to serve, and denying service should not be an illegal act as their prospective customers can go elsewhere to find another business to serve them, which is exactly what those disgusting hatemongers did.

You are of course referring to this: https://blog.cloudflare.com/why-we-terminated-daily-stormer/

Why try to hide behind your words? Just straight out say you are angry at CloudFlare for ceasing to host violent neo-Nazi propaganda.