How many of those employees are there?
In what fashion do you audit and track the activities of those employees?
What training are these employees given to avoid social engineering? What firm provides the courseware?
What's the escalation process for complicated, non-no-brainer reset situations? If a support person isn't absolutely sure whether they should reset something, how do they get a second opinion?
Are the support people who are entitled and able to make these changes incentivized to close tickets as quickly as possible?
Do you monitor "out-of-process" changes to recovery email and password settings, so that you can see trends over time and by particular staff members?
Has any third party security firm assessed your service recently specifically for this attack vector, for instance by conducting social engineering testing against your support staff? What's the firm?
How are you MINIMIZING, rather than just improving, the "human factors" involved in assessing whether accounts can be altered based on anonymous incoming callers and requesters?
This is a HUGE, TERRIFYING vulnerability. Email providers are the single most important security service people use; if your email is compromised, many (most!) of your other services are compromised as well.
Our absolute focus is on minimizing the human factors.
In the past year and a bit since that incident, we have improved our escalation policies and support training, as well as let some support staff go.
But more importantly, we now have an automated account recovery system which can be used to verify ownership of the account using a number of different factors (not all of which I'd like to talk about in public - again, if an attacker knows the full algorithm it helps them game it).
Questions like these are not unreasonable for a customer to ask a service provider with respect to identity management and protection of that customer’s proprietary and confidential information.
With respect to the first question “Exactly which employees have the ability to alter recovery email settings.” Not being able to have a prepared answer for this question suggests that you don’t have a formal policy or standard procedure around role based capabilities in your operation.
The second question is an extension of the first.
“In what fashion do you audit and track the activities of these employees?” Not being able to answer that question suggests that you don’t have an auditing process around employee actions with respect to account changes.
“What training are these employees given to avoid social engineering?” Not being able to answer this question suggests that you don’t have such training in place.
“What’s the escalation process for non-no-brainer reset situations?” If your processes are written down and staff are trained in them, a very simple description here would not create a security’s risk of any kind. Not doing so suggests that the process is not formally specified or is quite ad hoc.
“Are the support people who are enabled and entitled to lose the tickets incentivized to close the tickets as soon as possible?” It seems that your internal security posture would make that clear, and it is unclear how stating that correctness is more important than speed in user account modification poses a security risk.
I’ll pause here and summarize. Answering any of these questions is not going to pose a security risk unless such answers expose to your users reasonable measures that you are not taking or haven’t thought of.
Today, that role is granted to a much more limited set of senior security staff (currently 3 people). Regular support staff can not alter security-sensitive details about accounts. If your account is owned by someone else (e.g. family or business, or part of a resold package) then they can still alter recovery options, as they own the account.
In 2016 before we had automated account recovery, lost password was in the top 3 categories of ticket every single week! Every member of the support team dealt with multiple account-loss tickets per day, both forgotten password or stolen account.
Stolen account losses are way down now we have app passwords, we often only have to block a single app password and notify the user rather than locking the entire account. Forgotten passwords have not reduced, but most people are able to recover using the automated tooling.
---
In what fashion do we audit and track? A few ways - we log every API call at the lowest level. We log each override when the support person accesses user accounts against the ticket that they come in through, so we can see why they were accessing that user.
We could always do with better tooling to introspect logs, but the data is all captured and can be followed through after the fact. Support staff have no way to wipe their audit trail.
---
Training - in 2016 we didn't have much formal training for our support staff - they learned on the job from each other. We are very aware that this was a failing at that time.
We have more training now. We did a lot of work at unifying our support teams across the FastMail and Pobox/Listbox family throughout 2017, and that led to better training and induction materials, as well as better internal reference material for support staff to use.
Early in the induction process for all new support staff is a description of how social engineering works and a warning that urgency is often used in social engineering attempts, so when in doubt, slow down and get a second opinion (which leads to complaints about slow support, but that's the tradeoff here.)
---
Escalation process - as mentioned earlier, if you had 2fa enabled then it has always gone straight to the senior security team, which is based in Melbourne and consists of our most experienced and trusted people. Neil (author of the blog post this HN refers to) is of course one of these people.
With lost passwords no longer a highly common support request, all support tickets requesting manual account recovery are escalated to our senior team for review.
---
Support people have no incentive to close tickets quickly. Absolutely. That is a bad metric, and it's not a metric we have ever used.
Time to first response and time to followup responses are tracked, but there's no incentive to close tickets.
This answer is a no brainer and I should have answered it in the first response - sorry. I was still rushing through initial responses at that time, and there were too many points in that post to think about them all at once and still respond quickly. The real-time nature of this hacker-news medium encourages fast answers above complete answers. I hope this longer response helps clear up remaining questions, at least to those who see it!
---
There's another blog post coming soon about the account recovery system in particular, which addresses exactly how we've minimising human involvement in recovery decisions while not excessively punishing real human frailty amongst our customers.
Can I just clarify some things for peace of mind?
1) When you say regular support staff cannot alter security-sensitive details. How is that done? Do they only perform changes through a limited set of UI?
2) When you say if 2fa is enabled it goes to senior security team, is that an automated process such that support staff don't see that ticket at all? The support ticket interface doesn't seem to have anything that helps to automatically route password reset requests.
3) Was the security incident involving ghouse through support tickets?
4) Do the senior security team have direct data access? i.e. do they also change things through a UI or do they have capability to directly change data?
Thanks
The UI given to support staff doesn't have the ability to update security credentials for users because they no longer have the "can update security credentials" role like they did in 2016. I don't even have it any more.
2) front line support still see all the tickets first, and they route them as appropriate. Sure this takes longer, we don't have 24 hour coverage of senior security staff (not entirely true, we have 24 hour coverage for emergencies. Somebody forgetting their password is not an emergency in this context)
3) the security incident involving ghouse was entirely via support tickets. His description was accurate, front line support send the pro-forma "we need a bunch of these details", got back some pretty half-arsed details that didn't meet the bar of what was supposed to be provided, and helpfully made the change despite our policy. The helpfulness of humans is a major bug with any security system, and this particular human tried to be too helpful.
4) The senior security team also use a UI. Operationally, they all have the ability to write code that directly changes things under the hood, but that code also has an audit trail and goes through review. It's always quicker and easier to use the UI, so that's what they do.
The UI is not just available to those three people, it's also available to anybody who has a multi-user account and needs to administer their own users. It's still a standard part of our system, just restricted in who can use it at an "any arbitrary Fastmail customer" level.
I would like FM to provide something akin to Google's advanced protection program. Those of us who are careful not to lose our login credentials should not have to suffer a weak recovery process for the convenience of those who do. I personally would rather opt my account into a stronger recovery process even if I can't access my account for several days or a week or more.
Check out the longer response here:
Is it possible under any set of circumstances for your human employees to alter accounts? If the automated system fails, are accountholders out of luck?
In some cases we haven't had sufficient information on the account to ever verify that account's owner, and they never got their account back. Some users refuse to give us enough information to allow us to later positively identify them - so yes, those people will be out of luck if they lose their credentials.
I respect that, as CEO, you're genuinely responding to your customers in this thread instead of fobbing it off to someone else.
Hopefully, this can all be explained, resolved, and /or remedied in good time.
But I've started to think about moving back to Chrome for "high security mail".
My private mail is pretty bland and uninteresting, so I don't care too much about not using GMail there, but for my Apple account, Google account, Microsoft account etc. it might be a good idea to compartmentalize those "high value" things from everyday mail and go to GMail with the Advanced Protection Program (so no access from smartphone or iPad, I guess).
And looking at their web site I've learned that GSuite Business is affordable and allows adding domains hosted elsewhere. Good.
What do people think about this?
But then the next step: what about losing my domain? My registrar is a reputable German domain hoster, but certainly no Google. On the other hand, Google doesn't register domains, but has "domain partners" like "domaindiscount24" (that I've never heard of before), so I guess not much to win there.
How many engineering employees does Fastmail even have? How much better would each of them have to be than one of the best-paying security teams in the entire industry for them to match up?
I could go on, but to me, you don't really even have to think hard about this.
Does that make the claims that Protonmail is more secure than Gmail false? - https://protonmail.com/blog/protonmail-vs-gmail-security/
${All the things I said previously}. And, Google Mail is one of their flagship products.
Most of what is on that ProtonMail page is nonsensical. The claim that is relevant to the discussion here --- that ProtonMail has a "smaller attack surface" and is thus structurally more secure than Google Mail --- assumes significant facts not in evidence.
See downthread for my response to the claim that using a mail services outside the US somehow insulates you from NSA snooping.
But I think they're compromised by those same business models. Google wants to provide intelligence, and probably more important to them, marketing data. This requires that the consumer is an open book to them, and their business decisions incorporate that. Up until recently, they were actively scanning email for marketing insights. In addition, Google's operating complexity, both business and technical, increases the opportunity for failure. And their other business objectives compromise their security work. That's glaringly apparent for their Android platform. There's more surface. And in a Google world, the email account grants direct access to everything — location data, purchasing history, passwords, documents... everything.
For another dedicated email provider, what they have to protect is also simpler. There are fewer moving parts. There's less to protect, which means that there don't need to be as many engineers. That means a careful and well thought out email provider /can/ be as secure, by carefully limiting their exposure, doing one thing, and doing it well.
There's something to be said for careful application of open standards and open source software, a smaller and more responsive team, and not building a massive single point of failure. I am a current Fastmail customer, and hope to remain, depending on the outcome of this review.
Err... This could of been said about Apple Inc a few weeks ago then they go and have the root password issue.
Google were previously reading our emails for Ad purposes and some of their employees are still able to read our Emails, their privacy policy also indictates they will hand over our emails if requested by law enforcement which suggests it's weaker than protonmail.com end-to-end encryption:
> All emails are secured automatically with end-to-end encryption. This means even we cannot decrypt and read your emails. As a result, your encrypted emails cannot be shared with third parties.
If this is the case, how is Google being held as the Gold Standard?
A lot of this comes down to your threat model. If you are most worried about
Unless your threat model is "The NSA gives my hosting provider a court order" or "an employee of my hosting provider goes rogue", its pretty clear that GMail is categorically the best option. And in those two cases, its not clear that there are significantly better options.
[0]: https://landing.google.com/advancedprotection/
[1]:https://techcrunch.com/2017/03/24/what-to-do-about-those-gov...
I get and am not questioning that. It's just that your curiosity doesn't seem to have motivated you to do a first pass of, I don't want to call it 'research', but just basic poking around on the topic. You want links and info from some dude on the internet because what he says contradicts stuff you know from... something a vendor said about their product.
It's a totally sensible question but it's not some particularly arcane mystery to dig into. In tptacek's case, in a jiffy, you can bring up the 60-odd comments of his that mention 'Gmail' and get a reasonable idea of what he thinks of it and why. And if you think he's got it wrong, you can say, hey, tptacek, I think you're full of poop when you said [...]. And then maybe you can hash it out and one or both of you will learn something. But 'Citation, please', especially on trivially searchable topics mostly says 'I'm kind of curious, but I don't really care'. The person you're asking probably isn't going to care either.
If all we have are the same claim being repeated with the only way to learn about what makes Gmail the most secure email provider is having to trawl through 1000's of comments. It means Gmail is always going to perceived as more secure even when they may not be, because relatively no-one is going to trawl through 1000's of comments to make an informed assessment otherwise.
60ish is not 1000s. 69ish if you add the 9 about Protonmail. The guy posts on HN so much you can fairly safely go to https://hn.algolia.com and type author:tptacek [topic of interest] and find out what he thinks about it. If there was, inexplicably, a comic universe about HN mutants, he'd be The Citation.
This is getting weird. But I'll allow it.
This response says absolutely nothing about how the vulnerability is prevented in the future. It's just a bunch of vague promises and mumbo jumbo. What specific procedures are in place to prevent it? At a minimum, I expect to see something specific like when you guys almost lost your domain because of Gandi [1].
And even then, can I have an option to select absolutely no human intervention possible? Having any human intervention is simply not acceptable.
I already have multiple ways of recovering my account, and I never, ever want human assistance on this. I use a password manager, and I will never, ever need FastMail assistance on login.
[1]: https://blog.fastmail.com/2014/04/10/when-two-factor-authent...
Especially because technical folks like us don’t know to communicate well, words can be misinterpreted, etc. It’s actually not a good strategy to respond to such concerns in public.
Also in my opinion, people that make threats of leaving in public unless certain demands are met usually have their mind set already and can’t be swayed.
As for never needing human assistance, never say never — if relying solely on your password manager, I hope you have a digital last will for your spouse or children.
What this means is that if all recovery options are not working and I'm actually locked out, I can fix it.
I own the domain, I own my past emails, I can still get emails. Maybe I'll lose some emails for a day, but that's it. If I want to prove my identity to FastMail, I can also prove that I own the domain.
But the point is, getting locked out of something as important as email is not gonna happen due to my screw-up. It's more likely for a support loophole to screw me over.
Have you taken a look at the link I supplied above where FastMail wrote about how 2fa protection could be bypassed at Gandi? They were very specific and clear about the recommendations being implemented.
Now, compare that to their current response. I think definitely the difference can be seen.
This is serious stuff.
And, I'm absolutely serious about never needing human assistance. I already have mechanism setup for my family to retrieve my digital assets should I disappear tomorrow. I worked on this together with my wife. I know most people have not thought about this and you're right in your skepticism, but I'm serious.
But we technical folks are very odd and I’m assuming they have users that really need human assistance.
I do agree the response is insufficient, my point is that such discussions in public are dangerous for the company and it’s not the norm for company reps to give detailed explanations without prior preparations.
> I already have multiple ways of recovering my account, and I never, ever want human assistance on this.
Yep, I'll second this feature request. Put as many disclaimers and confirmation mechanisms on it as you need to in order to keep people from accidentally enabling it. I will happily assume responsibility for it.
That way I can setup my grandparents' phone number or something obscure as yet another recovery option.
And then, please let me lock down any possibility of your support staff screwing up.
It also brings issues of its own. Home phones are hard to block number on, and it could be used to troll people in the middle of their night. We need to consider those risks too - it's not a simple and obvious win.
Security paradigms have been steadily moving beyond a hard-boundary-soft-center, to a defense-in-depth, distrust-your-own-services model. I was alarmed to learn last year, for example, that you use OpenVPN with fixed symmetric keys (--secret) rather than TLS with any forward secrecy (--tls-auth) for VPN between your NYI and AMS datacenters. https://blog.fastmail.com/2016/12/19/secure-datacentre-inter...
Presumably, running datalinks like this means you would have to have perfect trust in your long term key management and rotation. Is that something you plan on improving in the future?
Similarly -- I stumbled on this entirely by accident after your blog post about moving datacenters -- that your head of security ops & infrastructure tweeted "I will probably root my phone soon because Samsung's emoji set is worse than not having convenient OTA updates" https://twitter.com/robn/status/919194089920311296
I don't want to conflate anything -- a tweet on an engineer's own time about their personal devices isn't by itself a security problem. But it does reflect on the security mindset. If you had a BYOD policy, and this phone did end up being flashed to Lineage and be 3 patch levels behind (esp with Android's track record of RCE-via-media CVEs), this could definitely become a weakness on your entire infrastructure, and thereby on all of us as customers.
This is the type of thing I couldn't shake after learning about it. Of course, trust has to be placed somewhere. You have to be able to place trust on your ops and your infrastructure, but that's also a process, not a checkbox. People and devices can be trusted a little less in the overall security system, to provide redundant security. Could you clarify your position on how your staff is trained about the human weak points, security as a lifestyle if you're security and ops, and how your security mindset incorporates defense in depth?
I’ve worked in companies with liberal BYOD policies for portable devices, but also tasted really restricted environments and such environments are basically highly regulated security theaters.
Users do stupid things of course and in corporations it’s worth it to restrict their devices, but restricting developers on what they can install and do on their own devices has a negative ROI and doesn’t go well. If you can’t trust a dev to manage his own phone, you can’t trust him to build your infrastructure either.
And yes, we make mistakes as we are only human, which is why a phone should not be enough to compromise that infrastructure anyway.
PS: your mention of that Twitter account is creepy.
With no context, I agree. But I'm not exactly stalking engineers here - there was literally a direct link to that twitter from the Fastmail updates mailing list that went out, when customers were notified of the NYI datacenter move. Made me do a double take.
Cheers.
You're right that security is a process. We're always working to harden and segment our internal services, as is best practice these days.
Ongoing professional development and training is important for our security staff (indeed, all our staff, because everyone matters for security). The security landscape is always changing, and it's not something that's ever "solved" - it's a situation to stay on top of.
There is no doubt that in this specific case our human factor screwed up, and I'm really sorry about that.
First I'm going to post the standard response that our team has written for any new support tickets that come in about this today, then write my own personal apology and response here.
---
Thanks for getting in touch with us about the report on Hacker News about our security procedures.
As we say in our recent post about security at https://blog.fastmail.com/2017/12/05/the-fastmail-security-m..., security is a process, not a checkbox. We do our best to be continually improving and upgrading our security procedures, and offering our security-minded customers the most robust, industry-standard options possible.
However, we have been less diligent about forcing older accounts to upgrade their security settings. With a range of possible security option states, customer support is occasionally placed in a position to make a judgement call. As the post indicates, the incident in question happened immediately after a major round of security changes. There’s no way around it; someone made an exception they shouldn't have.
Social engineering is always one of our biggest concerns. As any number of well-known break-ins have demonstrated, the "best" security hack is often to sidestep it. Since that incident, we have taken substantially more aggressive steps to close off avenues of attack and human review. We are constantly trying to narrow the number of accounts that even can go to a human for review, and for those that must go to a human to provide as much notice as possible to the account owner before possibly allowing the attacker to have access. For instance, some cases take 24 hours before the reset password goes into effect. If you are a legitimate account owner, this has the often frustrating side effect of locking you out of your account for 24 hours. But, if you have been attacked, this gives you the opportunity to keep the attacker out.
Thank you for sharing your concern with us, and I hope we’ve addressed yours.