jackcook

Thanks for reading! The two stories are of course deeply intertwined: we wouldn’t have found the new side channel without the cautionary tale about machine learning.

But the finding about ML misinterpretation is particularly notable because it calls a lot of existing computer architecture research into question. In the past, attacks like this were very difficult to pull off without an in-depth understanding of the side channel being exploited. But ML models (in this case, an LSTM) generally go a bit beyond “statistics” because they unlock much greater accuracy, making it much easier to develop powerful attacks that exploit side channels that aren’t really understood. And there are a lot of ML-assisted attacks created in this fashion today: the Shusterman et al. paper alone has almost 200 citations, a huge amount for a computer architecture paper.

The point of publishing this kind of research is to better understand our systems so we can build stronger defenses — the cost of getting this wrong and misleading the community is pretty high. And this would technically still be true even if we ultimately found that the cache was responsible for the prior attack. But of course, it helps that we discovered a new side channel along the way — this really drove our point home. I probably could have emphasized this more in my blogpost.

Websites doing this would have to be careful about it: they might become the only website triggering a lot of interrupts randomly, which then makes them easy to identify.

Our countermeasure which triggers interrupts randomly is implemented as a browser extension, the source code for which is available here: https://github.com/jackcook/bigger-fish

I'm not sure I would recommend it for daily use though, I think our tests showed it slowed page load times down by about 10%.

Thank you! Really appreciate it

Yes you're spot on, the nonlinearities come from the full Mamba blocks, which I left out of this post for simplicity/to focus on the bigger ideas the paper introduced. You can see it marked by the "X" on the right-most part of Figure 3 in the Mamba paper: https://arxiv.org/abs/2312.00752

Thank you for the kind words! I think it’s mostly to reduce complexity during training. Here’s an excerpt from page 9 of the Mamba paper:

“We remark that while the A parameter could also be selective, it ultimately affects the model only through its interaction with ∆ via A = exp(∆A) (the discretization (4)). Thus selectivity in ∆ is enough to ensure selectivity in (A, B), and is the main source of improvement. We hypothesize that making A selective in addition to (or instead of) ∆ would have similar performance, and leave it out for simplicity.”

Wow, it was quite the surprise to wake up to seeing this post near the top of HN! I wrote the post, happy to answer questions if anyone is wondering about any details.

Yes, you're right, I should have mentioned it in the post, but I used pure greedy sampling for the GPT-2 outputs since I couldn't do anything but that for the Apple model. So temperature was set to zero, and there was no repetition penalty.

I used greedy sampling (temperature 0) for all of them. Since I didn't have access to logits/probabilities for Apple's model, I wasn't able to do anything else in a way that would be fair.

Some context: In the upcoming versions of macOS and iOS, Apple is including a predictive text model which offers suggestions while you type, which they’ve said to be a "transformer model". I managed to find some details about this model, including details about its topology (which looks a lot like GPT-2) and its tokenizer, and I was even able to peek in and see several of its top predictions while typing!

Hopefully this can give some insight into some of the trade-offs that Apple went through to put a model on every iPhone and MacBook — it’s small, it has a pretty narrow scope, and it’s not very capable on its own.

Some context: In the upcoming versions of macOS and iOS, Apple is including a predictive text model which offers suggestions while you type, which they’ve said to be a "transformer model". I managed to find some details about this model, including details about its topology (which looks a lot like GPT-2) and its tokenizer, and I was even able to peek in and see several of its top predictions while typing!

Hopefully this can give some insight into some of the trade-offs that Apple went through to put a model on every iPhone and MacBook — it’s small, it has a pretty narrow scope, and it’s not very capable on its own.

I got a warning today when submitting my app that said I was using a private API function. Turns out they've essentially blacklisted certain function names that match anything in one of their private frameworks. Renaming functions (making a breaking change) is apparently necessary to avoid this: https://github.com/realm/realm-swift/pull/7951

I don't think any of these side channels are really easy to pull off without the technical capabilities of a nation state or something similar. I personally think embedding a malicious script in a CDN (e.g. https://blog.ryotak.me/post/cdnjs-remote-code-execution-en/) that serves a script for a large website, or something similar (https://blog.igorescobar.com/2016/08/21/ive-the-chance-to-tr...), is more realistic than getting the victim to install your program -- I would imagine sensitive individuals are very concerned about installing arbitrary software.

We did get a comment about this in our rebuttal but didn't end up including it in our final paper -- we found that we distinguished sites with the same frameworks (such as react, angular, and jquery) at the same accuracy at sites that used different frameworks.

We didn't do much research into content/non-homepage paths but it's a good area for future research. I would suspect it'll still do pretty well.

And yes, we concluded that the source came from interrupts (in Table 3 of our paper you can see we ran an experiment with frequency scaling turned off), which does make me question the practicality of hertzbleed. I wouldn't doubt it can be exploited somehow though.

You'd just need to put the script on any webpage the user might access and leave open, such as Google, or Facebook, or whatever. The attack isn't specific to JavaScript, so really you could put this in a desktop app too, think Slack, Spotify, etc. Any app or website that you know the target user is likely to open. CDNs are also a great target.

We evaluated on 100 websites as a proof of concept, but we also included experiments in an "open world" setup where the classifier has to predict whether the activity is from one of 100 sensitive websites, or whether it's none of them, and found that it's still very accurate in that more realistic setup. You would need to retrain to identify more websites outside of your set of 100.

The websites would need to be public, which is basically the same limitation as hertzbleed, since they need to know what they're looking for in order to identify an activity. Some use cases with this limitation aren't too hard to imagine: maybe you're in a country that bans access to major Western news sites but you're evading censorship with a VPN.

You can pull off attacks like this from JavaScript by repeatedly recording the time and training a machine learning model on traces of instruction throughput over time, which my group did in a recent paper: https://jackcook.github.io/bigger-fish/

We did exactly this in a recent paper we're presenting at ISCA next week (see https://jackcook.github.io/bigger-fish/) -- it's very possible for an attacker to do this. However, we didn't find that the signal the attacker found was due to frequency variations (and we did run an experiment to test this), but rather due to system interrupts.

Yeah we tested with only a couple configurations for the paper, and the model only took traces themselves as the input, no additional features.

In an earlier version of this project, I sent my code to 30 of my friends (all with different locations, networks, OS, etc) and had them collect a bunch of traces, and we found that the models lose a little accuracy when predicting traces on other machines, but not too much (maybe low 90s to low 80s, can be worse depending on the machines). We decided to focus more on the system interrupts/side channel analysis for this paper but we think that would be great follow-up work.

Yeah, disabling or suspending JS in background tabs would definitely kill this version of the attack. However, our attack also isn't limited to JavaScript -- many of our experiments in the paper are implemented with a Python attacker. It can be implemented in basically any programming language and embedded in any application on your machine.

Twitter thread: https://twitter.com/jackcook36/status/1534920169369309184

Paper: https://jackcook.github.io/bigger-fish/paper.pdf

GitHub: https://github.com/jackcook/bigger-fish

We found that machine learning can be used to identify activity on your computer from traces recorded in JavaScript that measure CPU instruction throughput over time. Please let me know if you have any feedback or questions! Really happy to discuss our work.