Unfortunately, there's not much slack left to cut. Linode pulled that line taught with the last major breach of their system (and their subsequent opaque response).
It's clear Linode's security practices leave a _lot_ to be desired, and their response to this incident, while better than the last, didn't go anywhere near far enough in terms of transparency (as well as demonstrating some fundamental gaps in their understanding of the current state of infosec).
If I can't trust you at your word, you no longer have the privilege of holding my data or my CC info. Two breeches with a lack of communication really does spoil overnight the trust that has been built up over years of wonderful and reliable service.
This security incident is very upsetting, but for me the linode track record in communication, responsiveness and support is still pretty amazing compared to the competition. At least at this price range.
AWS might be safer, but I basically don't get any support (for a reasonable price like I pay linode), Rackspace didn't seem nearly as responsive or transparent on much more trivial matters. Who else is out there which is worth switching to?
(I'm not being sarcastic, I'm genuinely curious)
I fail to understand how you can think Linode has a great track record. It is an unequivocal disgrace. TWICE they have mislead their own customers over a major security incident. And there have been lots of times during outages (in particular my time at Fremont) where they were MIA.
In an ideal world, yes, they should be more secure. However, as in this case, they got taken advantage of via a zero-day attack, with others planned well outside the scope of what Linode could have planned for. Which is insane. Can you even name something, anything that they could have done to protect themselves? Additionally, given the unique form of attack, figuring out what was going wrong was probably not possible. Thus, they knew as little as you did.
And then, everybody switches to some other provider. But do they switch to "super secure, we examine every byte of the software that we run to make sure we're bullet proof" hosting provider? NO, everyone just switches to another commodity VPS provider that is vulnerable to all the same super high level attacks that Linode is vulnerable (maybe even more attacks, given that Linode actually has a tremendous amount of experience).
In reality, you're only getting more security by switching to a less prominent hosting provider, A.K.A. security through obscurity. Which is the worst kind of security because it's not secure at all.
It's like getting mad at the mayor of your city when a meteor falls on your house: unproductive and misguided.
I'm just as annoyed at Amazon for this, to be honest, and in the large, annoyed at our industry for being so unnecessarily secretive. We need to stop thinking of our infrastructure as our competitive advantage; to pick on Google as an example, while Google are obviously masters of running systems at scale, their infrastructure efficiency is not the reason people choose Gmail. Obviously their platform gives them some competitive advantage but, for example, their policy of withholding even the innocuous names of internal systems is bizarre. I think the rest of the industry follows that lead.
It's weird that we embrace openness in the FLOSS communities but when it's time to build a revenue-making company, the details of the inner workings are immediately a hush-hush secret. If you're doing something simple enough that describing it means someone can replicate it, it's an idea that can be replicated trivially anyway. I bet everybody in hosting knows how Linode works, and I doubt there's any kind of espionage taking place.
In this case, it's fine to be secretive if you'd like, but at least tell me how you plan to prevent the problem from recurring. Linode always says "we're working diligently to prevent this from happening again" but provides no details whatever. The announcement from the founder of Linode[1] underlines this; the entire tone of the post is "here's how we band-aided the immediate problem," with no details on where they go from here as a business or culture.
When a security incident happens, I believe most security professionals would advise to keep details to the minimum necessary. I can imagine how misleading info can cause panic and dire consequences (to both linode and its customers). In Linode's case this could have been mandated by the FBI even, giving Linode no choice.
For me, linode is still one of the more transparent providers out there. I doubt AWS or any other provider would be more forthcoming if something similar happens.
Of course, there's a lot of security improvements to be made. I hope Linode would shake-up and improve and signs are they're doing that.
I'm still curious to hear some brand names that are better in that respect (hence my question about). From what I read there really is no better alternative currently at this price range.
I'm merely a hobbyist/researcher, so I don't have a production environment to worry about.
I need to be able to serve my customers, and they are all over the world. So I need a provider with data centers in multiple global locations, that offers an API, that has decent support that responds quickly, and at a similar price range... Interested to find alternatives that are also more secure and better at communication.
If they had access to the database, it may have been possible to delete malicious jobs from people's histories. Even if the user had email notifications turned on, an attacker with full access to the database could have turned it off temporarily (just flip a boolean flag).
This whole thing really makes me wonder how many hacks and deals behind the scenes are going unnoticed.
what's stopping the bad guys from just proxying dns queries they don't care about to the original NS?
with this sort of trickery you could get a "domain control validated" https certificate too!
My usual suspicion is that in general, the volume of DNS traffic should give you pause before you start putting custom code in the path of answering a query. Clearly it's possible -- Route 53 is built upon that very notion -- and I suppose in this scenario it's feasible.
Don't forget every Linode has a hostname under linode.com. I think splicing yourself in and running a conditional on every query would overwhelm whatever you point the firehose at and you'd have to plan accordingly. All it would take would be to add a couple hundred milliseconds of latency to the average DNS query (even before the inevitable carpetbombing of p99 latency) and a competent high-traffic administrator is going to start looking around.
Adding a conditional ("do I answer or do I proxy?") on every DNS query -- and there are many -- is going to introduce enough latency to be noticed unless you throw a lot of gear at it. And you're still going to introduce latency by inserting another hop. That's my point, though I do agree with you.
Welcome to the world of recursive name servers, there is a lot of software out there that does exactly what you just mentioned, I fail to see what would be hard about making this change.
Hm, why? Any modern CPU is blazingly fast. Writing it in Ruby probably wouldn't be smart, but Python + PyPI or Lua + LuaJIT would easily get within a factor of 10x of C.
This isn't a "could it be done?" exercise, it's more of a "could it be done without detection?" exercise. For this specific case, it's a pretty big risk.
As for detection, that was the reason I brought up CPU power. Modern CPUs are so fast that that it seems like this redirector would hardly generate a blip in any chart (such as top).
I don't care about proving anybody wrong. I care about filling my knowledge gaps. I.e. it's interesting to try to figure out how something like this would be detected in practice.
Who is going to notice increased latency in DNS queries? Most likely web developers. Nobody else I can think of would do (non-cached) bulk DNS queries to random domains and actually be looking for millisecond changes in lookup time. And those developers would have no insight to the DNS infrastructure serving requests, so they'd have no idea to contact the DNS admins to investigate. Even the DNS admins could be fooled before they contact network admins to do further research.
The bottom line is not "has DNS latency changed?", it's "has DNS latency become unacceptably high enough to force me investigate?" Unless it's becoming a problem, I think anyone would ignore increased latency because they have ten other work tasks to deal with.
You'd be surprised once you start working with larger, higher-traffic infrastructures. If our average external DNS query rises 200ms, my phone goes off. There's more slack on p99, but it's also monitored.
All of the timings for the various parts of a request to the system that I administer are instrumented from a small libcurl app running in multiple ASNs remotely, because Pingdom and other services do not provide the resolution that we need. They are then rendered on a stacked graph that always lives on my third monitor, and any significant deviation averaged out over five minutes catches my eye.
I know it sounds like overkill, but it's crucial at scale.
Think about the dumbest way you would do that. Then implement it. That's how simple our system is.
I don't understand why HSTS didn't allow some sort of pinning, or the ability to specify a certain kind flag in the certificate is required (like EV).
No, nearly ideal use. Its like a strategic nuclear weapon. You don't use it sneakily, that's the opposite of the whole point. Always intimidate as publicly as possible and in the tech community messing with linode is about as public as it gets.
The other part is showing off, its like declaring "we have access to them all but we don't care about name.com". Maybe they do...
Or, they're lying. I mean, they could be lying about the whole damn story, who knows (not me).
Assuming the account is completely true though, I would say they succeeded in showing off. It's pretty impressive that they took over name.com and discovered a CF 0-day just to get into Linode, all just to fuck with one IRC network no one cares about.
> All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.
Actually, the biggest lesson I'm taking away from this is to never trust any one piece of software, and to always have multiple lines of defense/alerting sitting on independent software stacks.
I wouldn't bet on that.
> There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet.
Isn't that the same logic Everydns/Dyn Inc. used when they censored Wikileaks?
> All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.
True dat.
I don't need to wager, and can speak with authority based on what I know (which I'd prefer to leave vague). There are two vectors into a Linode's filesystem from the perspective of an internal attacker: having root on the Xen host or gaining a login on the Linode. Knocking over the database and Web server gives you neither unless the person reused their account's password as their root password, in which case it's behind a cryptographic hash and subject to the typical rules there. If you own the database, you do have LISH access which gives you the equivalent of a VGA console; if someone left that console logged in, it's a vector as well.
The only vector HTP would have had in the general case would be bouncing the Linode. It's a fairly sufficient air gap, in a way.
ISTR being able to back up Linode images, migrate them around, and perform some other admin tasks from the web interface. So I still wonder if the "air gap" API is a bit more powerful than simply the ability to request a reboot. But, again, I don't know.
Without knowing what those things are, I'll just take your word for it that none of them could ever possibly be leveraged to compromise a host or guest without unmaskable and permanent messages appearing in the logs.
Don't we also have to take them at their word that they even had decrypted CC's? I haven't seen any proof yet that they obtained the decrypted private key, just their statement saying they grabbed in-memory keys.
So for the hackers to get the decrypted private key, then either Linode must have royally screwed up and kept the decrypted key in-memory during the rest of the month (which seems rather unlikely), or the hackers must have had control of the machine during the time in which they did billing (which I don't think is true, because billing presumably happens either at the start or the end of the month, and didn't the hack take place a bit earlier than that?).
So yeah, I believe them when they said they got the private key. But nobody's said anything to convince me that they got the _decrypted_ private key. And if the passphrase really is as long and complex as Linode claims, then it should be reasonably secure (caveat: I am not a security researcher, or otherwise qualified to judge the security of anything).
They bill you the moment you add a Linode, automatically, if your credit is not sufficient to cover the new Linode. Careful walking that assumption too far; I think it's safe to say the key was kept in memory.
Good move on their part.
Domains are jacked and proxied a lot more than people know. The hackers have custom tools (rather than Squid + BIND etc.) that perform these tasks and keep them hidden.
Even better, you could host spoofed DNS for a number of Linode's on a single small 128-256MB virtual machine. The infrastructure required is tiny. Definitely possible, definitely happens all the time.
Keep up with CVE's, don't provide a wide attack area (so lock down interfaces to your machine and don't expose much to the world), and keep blast radii as small as possible (so even if your machine does get owned, you can possibly restrict it so it doesn't automatically mean they gain access to other systems in your network.)
Oh, and model the threats to your network/application. Make sure you're securing against the right threat. As an example, anti-malware is wholly ineffective against social engineering - maybe it's more productive to train employees and make sure that each employee doesn't have total access to all privileged systems.
Damned near every exploitable security problem in an application can be traced back to this one simple rule. XSS is a common one. The ones that don't conform to the rule are rare and magical, like unicorns, and usually involve something exotic like hardware side channels. If you take input it will be abused, so plan accordingly. That occasionally manifests in unexpected ways like timing attacks, in which case an attacker repeatedly sends you lots of carefully-chosen input to deduce something based on how long it takes your code to reply, much like cracking a safe. It's a basic attack on a string compare, which is O(n). This simple code is vulnerable:
def price_multiplier(form):
if form.input == 'super-sekrit-coupon-code':
print >>browser, "100% discount applied!"
return Decimal('0.0')
print >>browser, "Sorry, that's an invalid coupon!"
return Decimal('1.0')
A timing attack would try to deduce each letter here, since the string comparison will short-circuit once it finds a wrong character. Using a simplistic model, say I sent 'a' through 'z'; 's' took 10 time units to respond, but the other characters took 8. Now I try 'sa' through 'sz', and go from there. As a thought exercise, try to imagine a few ways to prevent it, and consider the pros and cons of each; now you're thinking in the security mindset.Securing an infrastructure is another story, but the rule is applicable with the occasional clarification to everything there, too. There are domains of trust even within your own organization; malicious employees will try to harm you, as well, and you get little to no warning of those. Do your interns need root on machines containing customer data? Employees are users too.
I've seen that kind of timing attack discussed somewhere, and the solution there was to do some kind of byte-by-byte comparison that would always take the same amount of time. It makes sense.
I just want to know one thing: did or didn't they leak the credit cards?
Does it really matter? If your card was used with Linode, it should have been blocked by now anyway.
- HTP claims to have{, had} access to name.com, which Linode currently uses. This access enables an unauthorized party to update authoritative nameservers for your domain; i.e., if you host at Amazon, very likely your authoritative nameservers are Route 53 on your account. HTP would not have access to modify the zone directly through the registrar and would instead have to hijack the entire domain with a working, completely-transferred zone on their own nameservers. For this to go down entirely unnoticed is extraordinarily difficult. I won't say impossible, but damned close without a copy of the zone in hand and with Linode running AXFR disabled (you should be too). There are subzones of linode.com; they wouldn't have gotten them all, and it would have been noticed within minutes.
- In order to attack SwiftIRC, to get back at some script kiddies DoS attacking them after their last release (because you know, that's a good target to burn registrar access on and all), HTP decided to backdoor SwiftIRC via their nameservers which are hosted at Linode. That's not the same as the registrar nameservers discussed above, but is instead the DNS data actually stored on a Linode on SwiftIRC's account. They do not hint what they were going to do with it once they had hijacked the nameservers, and I will not theorize. I could guess, though.
- Before utilizing their registrar access (from the first bullet point) to hijack the linode.com zone and intercept manager logins silently by redirecting traffic via DNS -- also fairly difficult to pull off without a good linode.com certificate in hand, in terms of keeping the TLS session non-suspicious to a browser -- they instead discovered a zero-day in ColdFusion (Linode's stack) and got in that way. That's much quieter and much more likely to not be noticed. If we take the FBI's actions at HTP's word, the FBI was the only reason Linode was made aware of this outside of HTP's control; a DNS hijack would have been immediately noticed by Linode administrators.
- Knowing what I know (let's leave it at that), a successful exploit on Linode's ColdFusion stack entails a database of Linodes, DNS, credit cards, e-mails, addresses, and keys to decrypt the actual card numbers, and a lot more data. You have to decide whether to take HTP at their word that they deleted credit cards. Consider your credit card and all prior credit cards compromised if it were in the system before April.
- The access that HTP obtained does not, full stop, lead to root on Linode instances without at least one shutdown job or change of root password job showing up in your Linode's history that you did not ask for. Your Linode's root password is not stored in any Linode system aside from on your Linode itself. Your LISH password, as they say, is, and according to them is stored in plaintext; if you see things on your Linode's console (located under the Remote Access tab) that you did not type, that access was used upon you. If not, it wasn't. If you used the same root password on your Linode that you did for your LISH password, consider that password compromised. I'm suspicious of the claim that they rooted all those (assuming) customers without any of them noticing their Linode being rebooted to apply the new root password to allow HTP in, and I would read that as "potential access" instead of "access". They probably bounced some nmap.org servers to reset their root passwords -- a Linode system requirement -- without fyodor noticing. Which is interesting for a couple reasons.
- Also, the access they obtained does not lead to root on the Linode host fleet itself, unless they are holding back some extra access they obtained such as a shared password between the ColdFusion stack and administrator credentials for Linode systems, which I consider unlikely for a couple reasons. With several days to get familiar with the architecture, HTP could have used their database write access to do things on the hosts, but it's a fairly limited set of things. Dumping Linode's database is bad, but root on their hosts is far, far worse, and by indications, I don't think they got it.
- How does this relate to the Bitcoin hacks of yesteryear, you ask? The Bitcoin hackers probably got in the exact same way -- Linode hinted at a compromised admin credential, which is close enough to do everything HTP was able to do -- then shut down and reset root passwords on the Bitcoin Linodes they were after, which then gave them filesystem access.
So ends clarifications, thus begins conclusions:
- PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND.
- PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND.
- Linode added a feature that shoots you an e-mail when your Linode is manipulated in any way via jobs, such as resetting your Linode's root password (a la Bitcoin/HTP hacks). It's depressing they had to do this, but pay attention if you get the mail. External monitoring like Nagios that pages you when your server goes down is also a good idea, as long as it is hosted at another provider.
- EDIT: After reading the zine, yet again, /CFIDE is the vector. There's no excuse for not hiding your administrative tools, generally the soft underbelly of the whole smash, from the Internet. None. It's one rewrite in nginx. Match /CFIDE<anything> from the public, redirect to /. Done.
- EDIT: Again, after looking at how trivial the exploit was, it's probably time to reconsider using Adobe ColdFusion from a business continuity standpoint. Half Linode's fault for not hiding /CFIDE, half Adobe's fault for the engineering missteps that lead to this capability for a remote attacker. We should be just as hard on ColdFusion as we are on Rails.
- SwiftIRC is a den of inquity, up there with EFnet; if you run a hosting provider, think twice about permitting SwiftIRC anywhere near you. To reiterate that, Linode was a casualty of someone going after SwiftIRC. Delink their nodes, cancel them, and kick them to the curb if you're interested in preserving your business. Not worth the money. Same with damned near all the IRC networks except OFTC and, to a far lesser extent, Freenode. There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet.
- Registrars (and CAs, though that's outside this discussion) are the weak point in the entire system. This is not the first time they have been shown to be so. Linode could be Fort Knox of digital security but if name.com falls over, it's all over; that's entirely outside of Linode's, and your, control. Currently, the registrar market is heavily profit-centric and, personally, I think people spend far too little on a domain in the general case. I would happily pay a registrar a lot more money -- hundreds a year or more -- if their offering were run competently, as it is fairly obvious name.com isn't. Compare your hosting bill to your registrar bill; what's wrong with that picture?
- HTP is apparently fairly easy to troll into using valuable access for vengeance purposes. Shameful target selection and a burn of a good hack just to root SwiftIRC. That's like pissing in the ocean for a good time.
- Linode got railroaded here and the general reaction by folks is a little overdone. You know that's true when even the hackers' overview of the hack specifically calls out people bitching about Linode security on Twitter. All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.