News for hackers. Stuff that matters.

A comment on The Big Hack: How China Used a Tiny Chip to Infiltrate Amazon and Apple. Parent comment

Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by RL_Quine on Thursday October 04, 2018 @05:43AM (#18138411)
SuperMicro hardware has very extensive IPMI integration into the mothebroard, which amongst other things can take over and inject frames into the network interface, emulate a VGA device, talk to the CPUs serial lines directly, flash firmware, control the state of a number of physical devices- and this is what it supports just from the web interface it presents by default with the password "ADMIN:ADMIN". My money, based on experience attempting to harden their devices, is that any modification were injected into the IPMI hardware where most of this was already supported.

This stuff ends up being extremely difficult to disable. The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC invisibly to the OS and DHCPs on its own to open up an administration port, web interface, and some assorted call homes. You have to explicitly tell it to use the non-connected port, change credentials, and modify it so that it is not accessible within operating system as well. Hopefully while the machine is offline to prevented any automated scanning finding it within your network.

The number of times I'd end up nmaping our local networks and being able to remotely access production hardware with an interface that allowed me to reach this interface was maddening. The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely nu-authenticated avenues exposed. The course of action was always just to write off the hardware and bin it, because god knows what impact you could actually have using that interface.

[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by transpute on Thursday October 04, 2018 @05:55AM (#18138473)
OpenBMC (Facebook, Google, Microsoft, Intel, IBM and others) is working on open-source baseboard management software, https://www.linuxfoundation.org/blog/2018/03/openbmc-project...

> The organizations behind the new project each have already made substantial contributions to creating open source baseboard management controller (BMC) firmware. Now, working together, they will define the vision for a standard stack that can be used across systems and computing environments.

LinuxBoot and Open Compute OSF are working on open-source server firmware that can be measured on every boot and validated against hardware root of trust keys controlled by the server owner instead of the server OEM, https://www.platformsecuritysummit.com/2018/speaker/hudson/

[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by alfalfasprout on Thursday October 04, 2018 @11:09AM (#18140564)
Wouldn't help. The BMC hardware has direct serial access to CPUs and other hardware in the machine. Communication is unencrypted. A hardware modification attack wouldn't touch the firmware at all and could still compromise IPMI functionality.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by greglindahl on Thursday October 04, 2018 @03:28PM (#18142892)
Reducing the attack surface does help -- you're making perfection the enemy of the "somewhat better".
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by tomaskafka on Friday October 05, 2018 @02:39AM (#18146157)
No, against determined adversary it doesn't.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by transpute on Thursday October 04, 2018 @07:28PM (#18144542)
Even now, supply chain hardware modification attacks remind people of fiction. However, the number of people known to be affected by buggy BMC firmware is orders of magnitude larger, as described two comments upthread:

> The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely un-authenticated avenues exposed.

[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by Cieplak on Thursday October 04, 2018 @05:56AM (#18138479)
It sometimes feels like certain hardware protocols were designed to be insecure. I remember reading about IPMI issues back in 2013:

https://www.itworld.com/article/2708437/security/ipmi--the-m...

[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by RL_Quine on Thursday October 04, 2018 @06:36AM (#18138658)
SuperMicro hardware in particular always struck me as such. Asking users to pay a license fee to be able to update the BIOS on their devices (after paying tens of thousands for the hardware itself) is a kick in the teeth.

https://www.virtuallifestyle.nl/wp-content/uploads/2016/08/S...

http://www.supermicro.com/products/nfo/SMS_SUM.cfm

[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by detaro on Thursday October 04, 2018 @06:43AM (#18138690)
To be clear, this is only for the ability to update the BIOS over the BMC interface, not for BIOS updates in general. (unlike some other vendors, where you need a support contract to be even able to download the updates)

At least on some boards you can boot the USB drive image containing the BIOS updater through the BMC and do a remote update that way.

[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by bubblethink on Thursday October 04, 2018 @01:14PM (#18141737)
Supermicro doesn't publish changelogs for their BIOSes. Their disclaimer on their BIOS download page says, "don't update the BIOS unless you have to". It's a ridiculous stance for a server company. You buy Supermicro over Dell for the price. That's pretty much it.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by shaklee3 on Thursday October 04, 2018 @06:09PM (#18144118)
Price, variation, and time to market.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by RL_Quine on Thursday October 04, 2018 @08:23AM (#18139213)
For the models I was looking at a number of years ago, the options were through IPMI or via a USB Floppy Drive. Perhaps my memory is failing me here, but I seem to remember being quite enraged at the prospect and would have done a lot not to have to pay the license fee on principle.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by majewsky on Thursday October 04, 2018 @07:16AM (#18138850)
"Designed to be insecure" is probably unfair to the designers of IPMI. Security was just not as big a concern as it is today.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by pjc50 on Thursday October 04, 2018 @07:37AM (#18138949)
This is only really valid for protocols or products designed before the Morris worm of 1988. Anything designed beyond 2000 has no excuse for not thinking about internet security.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by vel0city on Thursday October 04, 2018 @11:14AM (#18140610)
Well, IPMI isn't supposed to be exposed to the internet. Best practices have you running your BMC's on a completely separate, highly locked down administrative network.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by dsl on Thursday October 04, 2018 @05:47PM (#18143945)
I did a security audit on a company that had a setup similar to this.

After popping an internet facing web server, I was able to compromise the IPMI system and use the management network to bounce around to any server in the enterprise completely bypassing all the firewalls and segmentation on the production network.

Management networks need rack level isolation.

[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by greglindahl on Thursday October 04, 2018 @03:31PM (#18142920)
Best practice would be for the BMC to not have access to the regular network ports when it has a dedicated network jack. All the ones I've looked at don't have any kind of physical interlock or switch, it's a software interlock.

Now even a hardware interlock could be subverted, but that's harder than sticking code in the bmc firmware, which does tend to get updated during the life of a server.

[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by parliament32 on Thursday October 04, 2018 @03:21PM (#18142817)
Well yes, typically you'd 1) configure IPMI to use the dedicated port, 2) put those ports into a VLAN with no outgoing internet access. But since this is BMC, what's stopping it from just using your management or production port to fire off its secret phone-homes and whatnot?
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by village-idiot on Thursday October 04, 2018 @09:19PM (#18144998)
Computer security as a field and area of concern is significantly older than IPMI.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by contingencies on Thursday October 04, 2018 @10:37AM (#18140251)
The problem with an IPMI BMC is that if you have malware that roots the OS from the BMC that in turn roots the BMC to reinstall itself from the OS, then you can never actually get rid of it. I actually opened a bug with a major clustering vendor in about 2012 because of this. Their response was a docfix.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by pmorici on Thursday October 04, 2018 @08:15AM (#18139156)
If people are interested in digging into how IPMI works every SuperMicro board I've ever seen uses some variety of ASPEED Baseboard Management Controller (BMC). Facebook uses the same chip in their open hardware projects.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by jiveturkey on Thursday October 04, 2018 @11:33AM (#18140827)
> The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC

cool, thanks for that info.

> just to write off the hardware

maybe you could just standup the mgmt network but blackhole route it at each switch port. The mgmt NIC thinks it's working properly but it can't talk to anyone nor can anyone talk to it.

at the expense of a dedicated switch.

[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by exikyut on Friday October 05, 2018 @05:08AM (#18146732)
Or, if the server has a free PCI slot, stick the cheapest possible NIC in that.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by codedokode on Thursday October 04, 2018 @05:59AM (#18138488)
Cannot it be physically disabled with a jumper or maybe by cutting some lines on the motherboard?
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by RL_Quine on Thursday October 04, 2018 @06:05AM (#18138517)
No. SuperMicro make some models which appear to be for government application which lack the ASIC completely, they were available on NewEgg with no further description of purpose at one point.

https://my.mixtape.moe/sorqql.jpeg

[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by wtfstatists on Thursday October 04, 2018 @07:08AM (#18138810)
But without the IPMI kernel modules loaded, IPMI is harmless, right ?
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by 0xfeba on Thursday October 04, 2018 @09:09AM (#18139521)
No, it boots prior/separate to the board itself. It's basically a mini-PC embedded in the board that has it's own CPU/Memory and tentacles attached to everything in the mainboard.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by pmorici on Thursday October 04, 2018 @08:32AM (#18139278)
No, the controller runs it's own totally separate OS, has connections to most/all the buses and is able to emulate devices whether you have drivers installed or not.
[ Parent | on HN ]
Re:The Big Hack: How China Used a Tiny Chip to Infiltrate Am... by detaro on Thursday October 04, 2018 @07:12AM (#18138829)
No, since it can configure the BMC, which works without the OS.
[ Parent | on HN ]