he thinks that conservative white men are holding back society from progress. 

I don't know about the race or gender part of that sentence, but the point of being conservative is to want things not to change. To say a conservative opposes progress is more of a definition than an accusation.