The mistake you're making is thinking of weights as an avenue to store training data. This is wrong. I've hopefully explained it more in another comment but weights/neurons are essentially configuration settings.