The islanders don't know that there are only blue and brown eyes. For all they know, their eye color could be green, red, etc. Hence, the logic only applies for the blue-eyed people.