It doesn't need to be fear, necessarily. They live on an island of 120 people with whoever did it. Like the guy or hate him, they're gonna be seeing him every week from here til the end. Why turn on him for some bloke who doesn't even have a house here anymore?