SuperMicro hardware has very extensive IPMI integration into the mothebroard, which amongst other things can take over and inject frames into the network interface, emulate a VGA device, talk to the CPUs serial lines directly, flash firmware, control the state of a number of physical devices- and this is what it supports just from the web interface it presents by default with the password "ADMIN:ADMIN". My money, based on experience attempting to harden their devices, is that any modification were injected into the IPMI hardware where most of this was already supported.

This stuff ends up being extremely difficult to disable. The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC invisibly to the OS and DHCPs on its own to open up an administration port, web interface, and some assorted call homes. You have to explicitly tell it to use the non-connected port, change credentials, and modify it so that it is not accessible within operating system as well. Hopefully while the machine is offline to prevented any automated scanning finding it within your network.

The number of times I'd end up nmaping our local networks and being able to remotely access production hardware with an interface that allowed me to reach this interface was maddening. The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely nu-authenticated avenues exposed. The course of action was always just to write off the hardware and bin it, because god knows what impact you could actually have using that interface.

OpenBMC (Facebook, Google, Microsoft, Intel, IBM and others) is working on open-source baseboard management software, https://www.linuxfoundation.org/blog/2018/03/openbmc-project...

> The organizations behind the new project each have already made substantial contributions to creating open source baseboard management controller (BMC) firmware. Now, working together, they will define the vision for a standard stack that can be used across systems and computing environments.

LinuxBoot and Open Compute OSF are working on open-source server firmware that can be measured on every boot and validated against hardware root of trust keys controlled by the server owner instead of the server OEM, https://www.platformsecuritysummit.com/2018/speaker/hudson/

Wouldn't help. The BMC hardware has direct serial access to CPUs and other hardware in the machine. Communication is unencrypted. A hardware modification attack wouldn't touch the firmware at all and could still compromise IPMI functionality.

Reducing the attack surface does help -- you're making perfection the enemy of the "somewhat better".

No, against determined adversary it doesn't.

Even now, supply chain hardware modification attacks remind people of fiction. However, the number of people known to be affected by buggy BMC firmware is orders of magnitude larger, as described two comments upthread:

> The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely un-authenticated avenues exposed.

It sometimes feels like certain hardware protocols were designed to be insecure. I remember reading about IPMI issues back in 2013:

https://www.itworld.com/article/2708437/security/ipmi--the-m...

SuperMicro hardware in particular always struck me as such. Asking users to pay a license fee to be able to update the BIOS on their devices (after paying tens of thousands for the hardware itself) is a kick in the teeth.

https://www.virtuallifestyle.nl/wp-content/uploads/2016/08/S...

http://www.supermicro.com/products/nfo/SMS_SUM.cfm

To be clear, this is only for the ability to update the BIOS over the BMC interface, not for BIOS updates in general. (unlike some other vendors, where you need a support contract to be even able to download the updates)

At least on some boards you can boot the USB drive image containing the BIOS updater through the BMC and do a remote update that way.

Supermicro doesn't publish changelogs for their BIOSes. Their disclaimer on their BIOS download page says, "don't update the BIOS unless you have to". It's a ridiculous stance for a server company. You buy Supermicro over Dell for the price. That's pretty much it.

Price, variation, and time to market.

For the models I was looking at a number of years ago, the options were through IPMI or via a USB Floppy Drive. Perhaps my memory is failing me here, but I seem to remember being quite enraged at the prospect and would have done a lot not to have to pay the license fee on principle.

"Designed to be insecure" is probably unfair to the designers of IPMI. Security was just not as big a concern as it is today.

This is only really valid for protocols or products designed before the Morris worm of 1988. Anything designed beyond 2000 has no excuse for not thinking about internet security.

Well, IPMI isn't supposed to be exposed to the internet. Best practices have you running your BMC's on a completely separate, highly locked down administrative network.

I did a security audit on a company that had a setup similar to this.

After popping an internet facing web server, I was able to compromise the IPMI system and use the management network to bounce around to any server in the enterprise completely bypassing all the firewalls and segmentation on the production network.

Management networks need rack level isolation.

Best practice would be for the BMC to not have access to the regular network ports when it has a dedicated network jack. All the ones I've looked at don't have any kind of physical interlock or switch, it's a software interlock.

Now even a hardware interlock could be subverted, but that's harder than sticking code in the bmc firmware, which does tend to get updated during the life of a server.

Well yes, typically you'd 1) configure IPMI to use the dedicated port, 2) put those ports into a VLAN with no outgoing internet access. But since this is BMC, what's stopping it from just using your management or production port to fire off its secret phone-homes and whatnot?

Computer security as a field and area of concern is significantly older than IPMI.

The problem with an IPMI BMC is that if you have malware that roots the OS from the BMC that in turn roots the BMC to reinstall itself from the OS, then you can never actually get rid of it. I actually opened a bug with a major clustering vendor in about 2012 because of this. Their response was a docfix.

If people are interested in digging into how IPMI works every SuperMicro board I've ever seen uses some variety of ASPEED Baseboard Management Controller (BMC). Facebook uses the same chip in their open hardware projects.

> The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC

cool, thanks for that info.

> just to write off the hardware

maybe you could just standup the mgmt network but blackhole route it at each switch port. The mgmt NIC thinks it's working properly but it can't talk to anyone nor can anyone talk to it.

at the expense of a dedicated switch.

Or, if the server has a free PCI slot, stick the cheapest possible NIC in that.

Cannot it be physically disabled with a jumper or maybe by cutting some lines on the motherboard?

No. SuperMicro make some models which appear to be for government application which lack the ASIC completely, they were available on NewEgg with no further description of purpose at one point.

https://my.mixtape.moe/sorqql.jpeg

But without the IPMI kernel modules loaded, IPMI is harmless, right ?

No, it boots prior/separate to the board itself. It's basically a mini-PC embedded in the board that has it's own CPU/Memory and tentacles attached to everything in the mainboard.

No, the controller runs it's own totally separate OS, has connections to most/all the buses and is able to emulate devices whether you have drivers installed or not.

No, since it can configure the BMC, which works without the OS.

The article says:

But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

To me, that makes it sound like they could download from a remote host and inject code and do literally anything.

If this is indeed the case, I'm surprised someone didn't catch something earlier when the device was calling "home" over their network. I'm wondering if China stole BlackRidge First Packet Authentication tech [1] to keep things dark. BlackRidge is... "involved" in IC and defense projects.

[1]https://patents.google.com/patent/US8346951B2/en

You're not the only one worried about Lenovo.

https://freebeacon.com/national-security/military-warns-chin...

Not the chip itself but there's an investigation which I've already submitted:

https://news.ycombinator.com/item?id=18144519

  5. When a server was installed and 
     switched on, the microchip altered 
     the operating system’s core so it 
     could accept modifications. The chip 
     could also contact computers controlled 
     by the attackers in search of further 
     instructions and code.

So, in typical vulnerability/payload/exploit fashion, the board's bus is vulnerable by default, because the chip pierces all the usual lines of defense protecting against network and operator I/O. It carries a payload intended to target very common features used everywhere commodity servers are used, one that likely listens for DMA traffic on the bus, and alters the signal stream, by escaping upon the occurrence of a magic sequence, and inserting its own signal, before resuming the authentic stream in flight.

The payload could be pretty small, since the server boards are likely using OS packages that match the chipset. This limits the software to a small set of well known targets, Linux, Windows, Apple. Target their kernels, and you only have to snip out a small chunk of bytes, and splice your own pre-defined package in. Splice in a miniature runtime, that operates a turing complete set of operations, and open up a listener that waits for network access, and now, your payload can enable arbitrary code execution, irrespective of permissions.

Now, to exploit, the payload needs to time the opportunity to splice itself onto the disk correctly. If certain well-known chunks of code will always exist in each given operating system, then with every disk access event one just needs to wait for the inevitable moment those magic system-specific bytes travel over the bus, in order to replace the known bytes with the poisoned modification. Events might target when the bytes are originally installed with the OS, or every time the OS reads those known bytes back into live memory, from any source.

The total payload package could probably fit inside a couple of megabytes, pack on a few more for the "listen & splice" part of the attack to round out the entire mass, and all we know how much data an SD card can fit into say... five grains of rice?

It's not particularly magical, there's consumer chips around which are not a whole lot bigger (though obviously in a more standard package). You don't get a lot of resources, but you don't really need it if all the other frameworks are in place in other software. If this sort of thing is something you can buy on Mouser for a few cents, the espionage grade material is probably an order or magnitude more higher quality.

https://www.microchip.com/wwwproducts/en/ATtiny4

For scale, this alone is about the size of a large SMD capacitor and would basically be lost in most designs today.

For another example, I have a couple of these which are a bit bigger but have an ARM SoC and onboard Bluetooth (with antenna):

https://www.digikey.com/en/product-highlight/t/taiyo-yuden/e...

That's a crazy module, I had no idea anything of that scale existed for Bluetooth radios.

Figure it's custom silicon, given the nature of the story, and "magic" in the sense of "magic number programming" to time the attack.

https://en.wikipedia.org/wiki/Magic_number_%28programming%29

For example, looking for ELF or Portable Executable headers, as a crude estimate to determine attack opportunities. In this case, the magic numbers would probably be more selective and sophisticated, but still have an aspect of hard-coded values, since we're talking custom silicon.

This sounds like speculation. I'm quite capable of coming up with my own unfounded speculation, but there is a real report out there with the actual details in that really needs to be made public, legally or otherwise. There ought to be a CVE about this. Where is it?

They didn't do anything to the CPU, what they did is the modchipped the line from EEPROM and the board management controller.

They probably found it out when they were repeatedly tried to reflash the BMC flash, and saw that checksums did not match.

That would make a lot of sense and would give the attacker a way to interface with all of the other hardware (network, disk etc.). Do you have a source for this information?

I looked up supermicro blade motherboards, and saw that the chip was right near the IPMI chip's line to spi flash.

And prior to that, there were already persistent rumors in the Chinese interney of certain Chinese mobos sending "weird garbage on ICMP," and "BMCs that somehow boot and work with their flash memory soldered off"

Remembering that, I might even suggest that this is not a modchip that does something with signal on the go, but just a very tiny flash chip that has the modded firmware.

Going further from that, to pack, say, 16 mB on a sandgrain sized chip, the densities need to be like that of best flash chips out there, which also means that they have access to last gen flash fab.

An update on that theory: AST2400 has option for two SPI memories, one main, one "recovery."

https://download.csdn.net/download/duanzhang512/10385038

The recovery overrides the primary if detected by default.

The place they put their "filter cap" is right on top the empty TSOP8 pad for the recovery flash. And they probably ordered the factory to sneak the traces just a little bit more, or put hidden vias under it, or simply had somebody very dexterous to solder it to pads with hair thin wires.

This is the most plausible theory I've read in this thread. Assuming the image in the article is a stock image (there isn't yet a clear image of a definitely compromised board), then the added part could simply be another TSOP8 Flash part. This implies the firmware to the AST2400 is unsigned (which it appears to be, as there's coreboot options for it).

That makes the whole thing gloriously simple. A part "stuck on" afterwards is obvious. A part fitted into a no-fit footprint after optical inspection is not, it looks exactly as if it was meant to be there.

To me it looks like that bloomberg intentionally photoshoped chip labels to avoid risk of upsetting their manufacturers.

This would hardly fit the description of a chip "smaller than a grain of rice", though.

Is this datasheet available outside of CSDN? Do you by chance have a copy you would be willing to share?

eh, does this mean that the motherboard is not tampered with ?

Well, it means that provision for the second flash was already there, and PLA simply exploited that fact that Aspeed chips are virtually omnipresent in higher end servers.

It also means that the extend of intervention into board design was minimal, and that a trivial automatic xray would not have picked it up. And as implied in the article, later they buried the bug to beat the AOI, if it was done higher upstream.

So, they would've been screwed even if they were doing board testing outside of China.

That's a clever trick.

But the sole fact that the chip has "to phone home" makes detection trivial, and puts the usefulness of the method to nil - anybody sees the router blink when it shouldn't and your bug's cover is blown.

A photo of such a motherboard with a big arrow pointed at the additional chip would be a useful addition to this discussion.

Supermicro 6128 aka x10 series microblade. Those were very popular among Chinese DC operators during Broadwel era.

https://www.itcreations.com/dist/landing/i/MBI-6128R-T2/MBI-...

Left of the sata connector. An empty space with 8 pads for an smt eeprom or flash. It is occupied by the thingy on bugged boards.

Right below is the Aspeed chip - the BMC

Is there a list of known compromised Supermicro SKUs?

There really, really needs to be. If this story is real and things have been known compromised since 2015, it’s unacceptable that this information isn’t shared more widely. The “we’re going to follow them” idea doesn’t hold water over 3 years, countless small vendors using Supermicro and not being well-connected enough to know (like Apple or Amazon) about it.

No

Hm, but DMA messages get distributed over a parallel bus and this chip seems to employ a serial interface, so I would assume that it's not directly connected to anything that requires high throughput (i.e. memory, disk and peripheral access).