If this is the case, and I believe you, then this implies: "bytes are NOT all you need". You need a good representation of your image for the network to work well.

This is probably what you would classically call "feature engineering"? And this is all hyperparameter search stuff in a way.