But the previous consituency (Wansdyke) that covered a similar area was Labour from 1997 and until 2010 (the Blair/Brown years), when it was abolished and replaced by North East Somerset, at which point Rees-Mogg won it for the Tories. Prior to 1997 it was historically Tory.
So, we might conclude that it's a moderate Tory seat that doesn't mind going Labour provided its moderate Labour and not Corbynite-type Labour. It's certainly not populated by bunch of dyed-in-the-wool intransigent Tories, so there's every chance Rees-Mogg might have lost if the constituency didn't like him.
Interesting, Rees-Mogg is an arch-Brexiteer but his constituency only very narrowly voted in favour of Brexit (51%), so perhaps they like him and not just the party. Although the Labour and Lib Dem opponents split the Remain vote 50/50, which certainly helped him.
doing a good job should be more important than how one appears, however people's tend not to be rational.