There is evidence that CNN's use texture features more than shape features, i.e. have a texture bias. It's hard to tell in this case without access to the data/model, but it's very possible colour is being overvalued by the classifier and causes the errors.