Neural networks do generalise well, and their vulnerability to crafted attacks is unrelated to generalisation anyway.

Additionally the size of the network isn’t the same thing as GPU memory and you don’t fit training examples in memory anyway (?!).