I cannot speak for the industry as a whole, but remote management systems like this are bound to be common; any large enough physical datacenter is going to need a more efficient way to access a misbehaving system than sending a tech physically running to the box to plug in a keyboard and mouse. It should be extremely uncommon to have these management interfaces open to the public though, and I'll bet that's what NordVPN is surprised by. Generally these systems should be private and isolated due to the power that an attacker can wield through them.

Cheap datacenters are favored by VPN providers for their unlimited bandwidth and lax abuse policies.

Many of them allow access to IPMI only over a VPN, but do not isolate each customer’s IPMI to a customer VLAN. I personally know at least three large budget datacenters which allow all customers access to each others’ “private” IPMI IP addresses.

The higher end ones are usually newer-ish, but there are lot of older "computer rooms" that offer acceptable-level benefits for a reasonable price. Lots of legacy customers and ISPs in these rooms, for the record. You get what you pay for but a lot of older DC spaces are just fine for most users; everyone thinks they need 99.999 but most don't.

There are also re-sellers and managed services companies who take out a footprint in data centers and then lease space in their cabs, sell bandwidth, IPs, etc. Using a series of resellers you can often get around restrictions as to what you're doing -- small fry MSPs don't ask a lot of questions -- but still get a data center footprint. These are sometimes one-man shows, and their quality and professionalism are often sub-par, which is how you end up with default iDRAC creds and the like.

Check out Data Center Maps or WebHostingTalk for some examples.

Source: data center ops manager for several different companies.

But, yes, remote management is pretty common in datacenters. The fact that NordVPN wasn't aware of them just shows incompetence.

This is the dumbest thing I've ever seen... unless your firewall is between your host versus every other host and there's no multi-tenancy, this will suck.

Never trust the network.

There were many IPMI/iDRAC/etc. exploits published in the past few years. Throw a dart at a list of them, and you'll probably find one that was unpatched in most systems as of March 2018.

In secure environments one pulls IPMI module from the server or only uses the modules that have their own dedicated NICs that have to be wired to their own management network.

There also quite a number of sysadmins that connect idrac's to the "regular" network, instead of the sysadmin VLAN ...

Now wait for the machine to get rebooted (or do it yourself using the BMC, e.g. 'racadm serveraction powercycle' for Dell/iDRAC machines).

Even SecureBoot won't help as you can just turn it off using the BMC.

See here for a bunch of examples for Dell machines using the BMC's HTTP API:

https://github.com/dell/iDRAC-Redfish-Scripting/tree/master/...

Power failure would require both of the power feeds in the DC failing simultaneously and would be easily verified by contacting the DC and asking if they had any power outages reported at the time. Of course there are cheapskates who don't go for redundant power supplies so it's possible but would be indicated in the IPMI logs

I've worked in production environments with thousands of machines and random reboots are a completely normal event for some workloads. Combination of hardware issues & kernel issues with hundreds of thousands of lines of code makes it inevitable. I would be surprised if NordVPN even noticed and their architecture wasn't designed to automatically start everything at boot.

You can't be perfect at scale - you just need to design your work loads to be redundant and fault tolerant.

""All servers we provide have the iLO or iDRAC remote access tool, and as a matter of fact this remote access tool has security problems from time to time, as almost all software in the world. We patched this tool as new firmware was released from HP or Dell.

"We have many clients, and some large VPN service providers among them, who take care of their security very strongly. They pay more attention to this than NordVPN, and ask us to put iLO or iDRAC remote-access tool inside private networks or shut down access to this tool until they need it. We bring [iLO or iDRAC] ports up when we get requests from clients, and shut them down when they are done using this tools. NordVPN seems it did not pay more attention to security by themselves, and somehow try to put this on our shoulders.""

Even a lot of VM/cloud systems have some kind of virtual management console (Linode has their LISH system that lets you SSH in to console and Vultr/Digital Ocean have similar web based consoles .. AWS surprisingly doesn't. You can get console output but can't send VMs any console input).

Not only should have NordVPN been aware of this hardware KVM, they should have secured it and had version checks on its firmware as an essential part of their security. I could see this oversight with other companies, but not with one whose primary business claims to be security.

Power on/off should be done via APIs that issue commands to a PDU, like Atlantic.net started doing in the early 200s.

And there's nearly zero reason to access "console" - configure your server to always but off PXE and fall through to disk if that intercept is not needed.

Why is this surprising? AWS seem to know what they're doing in general, and this is obviously the right policy in this particular area.

They talk about the AWS “Metadata Service” Attack Surface and how juicy a target it is. I was just providing support for that opinion.

This is like an author of a website vulnerable to CSRF (because it relies on IP for auth) blaming browsers for allowing cross site requests instead of require proper authentication. Except that Amazon is powerful enough to get away with pushing all the effort onto developers and admins.

I guess a machine-local service that takes ownership of the metadata service and implements additional restrictions (such as limiting access keys to privileged users) might be doable.

True data centers where you own the hardware shouldn't... they give you an ethernet cord and everything is on you.

Absolutely. We had similar situation with one of the DC vendors.

If you're lucky during network install, you'll never need out of bound access.

But rebooting is still a manual process.

However, if the attacker gains RCE many IPMI implementations theoretically allow for DMA, but this is a significantly more complex attack to mount in practice with no public PoCs available.

- Or, request a private network with no connectivity and a VPN device connected on that private network.

- Or, hardware disable LOM.

There's no reason you shouldn't request an ASA from your datacenter provider in this day and age.

Most (not all) servers have out-of-band management; of which IPMI is just one of many such solutions.

It's also worth noting that the hack could have been against in-band management if the Nord used an OS image provided by the DC hosts. However OOB feels more likely given their description (as vague as it was).

One presumes that because of NordVPN's business, they're colocating a server or two in many very many "POPs", presumably not all of them have tight controls on physical access. Its likely that there are none available in many areas where they seek to maintain a point of presence.