Re:Attacks against machine learning – an overview
by joewee on Sunday June 10, 2018 @04:50PM
(#17280081)
Very well said, never thought about it in this way. It also nullifies a lot of propriety risk scoring models, like credit scores. I wonder what research is done around this for automated trading systems? I see an “attacker” that creates models who’s only purpose is to force another financial institution to make unprofitable trades based on reverse engineering the other traders trading modes. Eventually, if not already happening, trading becomes machines attacking other machines.
Re:Attacks against machine learning – an overview
by tzahola on Sunday June 10, 2018 @05:10PM
(#17280152)
>Eventually, if not already happening, trading becomes machines attacking other machines.
Welcome to high frequency trading. You’re a bit late to the party though (around 15 years).
Re:Attacks against machine learning – an overview
by robertk on Monday June 11, 2018 @07:21AM
(#17283350)
I believe you are wrong, if I am to believe my trader friend to whom I showed this thread and answered:
“Trivially true in the 'of course does behaviour of others matter' sense and in the 'could my actions influence others'. Not necessarily operational though.
Fine line from there to 'spoofing' (== placing trades solely with intent of engaging others to trade at price level) -- with is VERY EXPLICITLY not allowed and for which you can get fined and go to jail. Recall the case of that poor SOB out of London who was made a poster boy for the flash crash?”
It seems there is regulation against this.
Re:Attacks against machine learning – an overview
by pwaai on Sunday June 10, 2018 @04:11PM
(#17279933)
could you place a 'backdoor' in these neural networks? Releasing a publicly beneficial AI while allowing it to be activated like a Manchurian candidate.
Re:Attacks against machine learning – an overview
by mholt on Sunday June 10, 2018 @04:23PM
(#17279980)
That's basically what adversarial inputs are; except that those are unintentional. A true backdoor would probably look more like a ML system trained to give a nefarious output for the "backdoor" input (but this is like poisoning), and then whatever program relies on the output of the ML system would handle that accordingly; not unlike backdoors in conventional software. The output of a neural network is only as useful or effective as the software (or person) that makes decisions based on it.
With powerful machine learning systems, we need to think about security a little differently. See especially the section 4.8 about function approximation:
> Given a task for which no discrete algorithm is known to solve, there is a good chance a neural network can at least approximate it. The extreme value of neural networks are their ability, in many cases, to act as an unknown function that can map inputs to outputs with good enough generalization almost as if the actual function was known. This makes any system that relies on the difficulty of implementing an unknown function vulnerable to the malignant use of neural networks