> ...every hardware pentesting shop will be going after these boards like they're looking for golden tickets.

The way the original story was written, it suggested that four subcontractors were identified, and almost 30 targets selected, with the implied suggestion that either the boards were custom special order boards, or destined for a specific lot order made by a customer. If true, then it is unlikely these boards will be found in the wild except by slipping through those almost 30 targets into the used market.

Supermicro has about 600 board SKUs, so finding these needles in the haystack is likely more feasible by approaching data centers offering to help find the boards for free, in return for allowing the pentesting firms to take physical possession of such found boards.

The story did reveal that the original "tell" that gave away the chip was odd but not obviously malicious at first glance network traffic, and that the suspected intent was implementing an Advanced Persistent Threat model. The article also mentioned the chips were connected with the BMC, but it wasn't specified in the article whether or not the chips got out onto the Net.

So finding the boards in the wild probably will focus upon finding them in the same manner, over the network. Power down the server, let the BMC stay powered on, and watch for unwarranted network activity. Or boot a Linux on a stick that deliberately does as little as possible and premises networking allows it to do just enough routing out to the Net to capture traces of what unauthorized network traffic is trying to do, and watch for unwarranted network activity, in case the chip design is clever, and hides its activity until it detects the mainboard is already running before trying to inject its network payloads onto the mainboard's network interfaces.

What are others' thoughts on how to find these "golden ticket" boards?

That's the interesting bit. Having done so much to hide the exploit your most important aim is to hide it's presence. Anything that would just "connect to a Chinese server" would be discovered immediately. If the Bloomberg story is true, the network traffic scheme must have been extremely sophisticated to fool so many network security specialists at top companies for such a long time.

This, or the story is false, purposely or not.

I'd appreciate a detailed explanation of the steps needed to get from putting this chip on a motherboard to actionable intelligence useful to a hostile nation state.

If we're talking about being able to make changes at the OS level, surely those should be relatively easy to spot?

If we're talking about copying packets, wouldn't that be useless under most circumstances?

If we're talking about doing whatever on a mass scale, surely most of the data would be junk and a huge and very clever backend would be needed to sift out the useful elements?

Presumably none of this is impossible, but I haven't yet seen a good description of how it's all supposed to fit together.

Something like triggering a memory-read of a specific address the moment you see a TLS diffie helman exchange on the wire? The idea being that you can recover the session key, and thus break the encryption retroactively.

That requires more than just access to the network, but it is rather simple.

With just access to the network, perhaps one could do spoofed DNS responses that are never seen on the network? A very simple '1% of the time, set the gmail.com A record to the chinese gmail IP' might be enough.

One could create mayhem by sending false ARP or DHCP responses, but that is only mayhem. Perhaps if it is externally triggerable it is useful offensively as DoS.

Most places I know of isolate their OOB management network, requiring a vpn or jumpbox to access it. However, if someone did let their OOB network full outbound access, I could see this slipping through. I could imagine that simply going to a CDN or cloud provider like AWS/Cloudfront/cloudflare/akamai with a dns lookup along the lines of updates.supermicro.cdn-front.com wouldn't be too suspicious. At that point, you'd be looking for dns lookups and not firewall hits.

If you are blocking outbound, I could still this going unnoticed if you're not actively reviewing denials.

But, if you are properly watching dns lookups from OOB and it's anything other than necessary services (ntp, ldap, syslog), then this would get picked up pretty quickly.

> However, if someone did let their OOB network full outbound access, I could see this slipping through.

Quite, especially in small networks and inline ilos, entirely possible that people plug the ilo (ipmi etc) to a more open network. Sure, nothing in, but no block on stuff going out.

Sounds like many things would have to go right in the defender's court. Optimism is not a good defense strategy :)

Heh. One colleague has "Hope is not a valid deployment strategy" as a signature.

Most places might be blocking this type of activity by default. For most of our security audits, it's just assumed that the SM IPMI or Dell idrac is vulnerable to one exploit or another. We mitigate that by controlling the traffic. I feel this is common practice in most places that understand vlans and firewalls.

However, while blocking is easy, being aware of something like this is on another level altogether. Unicorn jumping over a rainbow level rare. You really have to be logging outbound attempts and dns lookups. Where I work, there is a full security team and they are at an insane level where they log the allowed traffic. One told me that the allowed traffic is more interesting than the denied traffic. Denied just tells them what we anticipated, while active helps them establish a pattern and look for deviations.

That's my point. Security people were analyzing network traffic for decades trying to spot something that doesn't fit, host-wise, pattern-wise or even packet-wise (see The Museum of Broken Packets[0], for example). And someone managed to somehow hide all this traffic from security experts working for Amazon and Apple, for months or years? I'm very curious to see how.

[0] http://lcamtuf.coredump.cx/mobp/

I think you misread - "Unicorn jumping over a rainbow level rare" was about catching it, not missing it.

That is a good question, did the Chinese target specific customers, identifying the relevant sub-contractors and modify these boards?

Or are these boards commercial-of-the-shelf things that are mass produced?

In the latter case these boards can just end up anywhere, in the first case not so much. As a result using commodity boards would provide some degree of protection against such an attack as slipping a manipulated one into a specific target's servers would be a game of chance. Also the risk that the attack is discovered would be much higher.

> That is a good question, did the Chinese target specific customers, identifying the relevant sub-contractors and modify these boards?

Why not go read the story? It addresses this question and many others HNers are asking now. It's actually a fairly detailed story, although annoyingly short on technical detail (which could have been because their sources refused to go into much detail).

I did read it, so if it was mentioned I missed the part where Bloomberg stated if it was targeted at specific orders and thus customers or if it was seeded into the supply chain to see what sticks.

So, if you know the answer, why not just providing the information? ;-)

I think the Command and Control traffic will be much harder to spot. In fact it might be dormant until an operative is on the same network and wakes it up.

This isn't how gag orders (or, in the case of online services, NSLs) work. A gag order can compel you not to share data, but they cannot compel you to lie, let alone lie elaborately, as Apple and Amazon at this point would have to be for their denials to be false.

If Apple is lying here, they're lying because they want to, not because something is forcing them to. I believe them.

"Did you receive an NSL?"

You have been compel to lie, by the government, to answer "NO" if you've received one.

If you can be compelled to lie that you have received an NSL then I do not see why you can't be forced to lie more.

We already compelled speech, why not more?

I thought that you could simply not answer the NSL question (hence warrant canaries)

Iirc that's only in theory. I don't think there has been any precedent set. But I always see people say that a warrant canary is a false sense of security.

> This isn't how gag orders (or, in the case of online services, NSLs) work. A gag order can compel you not to share data, but they cannot compel you to lie, let alone lie elaborately

Is there case-law to support this? I've heard the argument often, but I don't believe it has yet been tested.

It’s a well established doctrine that this is a direct consequence of First Amedment: https://mtsu.edu/first-amendment/article/933/compelled-speec...

I think there is no way they would claim they are not under any gag order if they were forced to never confirm such an order, if there also is the option of not addressing it at all.

  I think there is no way they would claim they are not
  under any gag order if they were forced to never confirm
  such an order

I've always thought such denials are pointless.

You think everyone at Google knows every secret gag order they're under? Of course not, that would get leaked within minutes.

If I was the NSA I'd find a "patriotic" mid-level or junior employee with the power I wanted to subvert, and give them a gagging order that stopped them telling their boss.

That way the Google PR can honestly say that the CEO and legal team haven't seen or heard of a gagging order.

I think it would be odd then if no one else in the company is aware of the issue that those mid/junior level employees would then leak the information to Bloomberg during an active investigation.

> Either way, the proof is in the pudding: every hardware pentesting shop will be going after these boards like they're looking for golden tickets. Either we'll get our die shot, or Bloomberg's getting their pants sued off.

From the story, this seems to have happened several years ago.

I wouldn't be surprised if house cleaning had been done for some time.

If the story is true then I doubt that the Chinese would have flooded the market with these boards. It must have been relatively targeted.

One thing that sticks out to me is that they are fairly persistent on using "Apple" rather than something like "Apple and its security partners and subcontractors", which would eliminate most of the potential for doubt.

> Bloomberg is exquisitely specific and detailed about the chip, including photos and placement (on the baseboard management bridge, disguised as a shielding element.)

They are very specific, yes. They show pictures of the alleged chips that are allegedly disguised, yes. They describe placement, yes.

However, photographic or video evidence of these chips on SuperMicro boards is conspicuously absent. The only visual of placement is an illustration, not a photo. I’m not suggesting Bloomberg is wrong. I find this story fascinating and incredible (if true). But I noticed there were no photos/video of the chips on boards.

The last paragraph of the press release on this hacker news post:

"Finally, in response to questions we have received from other news organisations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations."

So you're basically implying that Apple is colluding with the US government to actively lie to discredit this story.

The funny part is that, based on your tone, you can't imagine a case where: 1) that's possible and 2) Apple may not be a willing colluding party.

I can't imagine a case where it's within the US's power to force Apple to falsely deny a story for them, but not within their power to feed a fake story to Bloomberg.

The proof of the pudding is in the eating. That makes sense. I ate the pudding it tasted good it proved to me its deliciousness. There is no proof in a pudding that is not eaten.

You clearly not an Agatha Christie Fan