EvilModel: Hiding Malware Inside of Neural Network Models [pdf]

Re:EvilModel: Hiding Malware Inside of Neural Network Models... by blunte on Tuesday July 20, 2021 @12:26PM (#27895835)

Not what the paper discusses, but the title gives me a thought: can you train a neural net with a specific, predictable behavior that would generally go undetected but which you could use to your advantage? I suspect you could.

Imagine building an apparently useful trained model for finance which you have a special way to predict the behavior of and profit from. (This could also be likened to formal human finance education, which certain elite groups take advantage of, but I don't mean to go into conspiracy theory territory...)

[ Parent | on HN ]

Re:EvilModel: Hiding Malware Inside of Neural Network Models... by xyzzy21 on Tuesday July 20, 2021 @12:38PM (#27895990)

You mean like a "mole" or "sleeper cell"? :-)

Just give it the right input and it turns on its masters.

[ Parent | on HN ]

Re:EvilModel: Hiding Malware Inside of Neural Network Models... by blunte on Tuesday July 20, 2021 @12:51PM (#27896138)

Yes, basically.

Imagine in financial markets if you trained a system to identify trends in currency pairs (for example). Generally they would be benign and useful, but if the a certain input were present, they would advise an action which the designers would expect and profit from. The users of the system would follow the advice as normal and probably not even know that someone essentially had inside information.

[ Parent | on HN ]

Re:EvilModel: Hiding Malware Inside of Neural Network Models... by kiwih on Tuesday July 20, 2021 @02:38PM (#27897485)

There is a name for this in machine learning, it's called a 'poisoning attack'. Essentially you add false data into a training set. An example: https://arxiv.org/abs/1703.01340

[ Parent | on HN ]

Re:EvilModel: Hiding Malware Inside of Neural Network Models... by 123pie123 on Tuesday July 20, 2021 @12:39PM (#27896005)

Neuralnet/ ML version of "Order 66"

[ Parent | on HN ]

Re:EvilModel: Hiding Malware Inside of Neural Network Models... by blunte on Tuesday July 20, 2021 @12:56PM (#27896214)

If I read the back story on this correctly, then no - it's not what I mean. I don't mean to change an outcome, but rather to create a situation where you can be certain of what the oucome will be. Then you can profit.

Granted, I wouldn't do this myself as I do find ethical problems. But its a fascinating thought experiment.

And if you believe in human legacies, then you don't need AI. You just build a system and manage the players so that your offspring or their offspring can take advantage of the game you have setup. Maybe this falls into Rothschild-like territory (which I frankly don't believe humans are meticulous, lucky, and coordinated enough to pull off).

[ Parent | on HN ]