Colleges seeing a decrease in enrollment and "no one will want to go" are completely different things. If colleges are worried about it they probably have some market research to back up their worrying. Whether it will end up being true is a completely different story. However even a modest decrease in the percentage of the population going to college would have a huge effect on the money available to universities.