Someone just published a new attack at ICLR that does not even need confidence scores but works solely on the final decision output of the network and works as well as the best known attacks (that so far required gradients): https://openreview.net/forum?id=SyZI0GWCZ&noteId=SyZI0GWCZ