I'm not sure you can get much more "middlebrow dismissive" than assuming that people objecting are white and the people they are objecting to are black and therefore it must be for similar reasons to why white people objected to black people in the past for bad reasons, so you can just ignore them.

Importantly, even if you're right in a lot of the instances, it doesn't actually move the discussion forward in a useful way because it paints with too broad a brush and encourages others to do the same.