We can turn this around though, can't we? If the ISPs have no designs on violating network neutrality then why do they oppose it?

If you repeal the law against burglary and then burglary doesn't immediately skyrocket, would you say that we shouldn't have a law against burglary? Of course not, because regardless of how often it happens, you'd like it to never happen and would want to prosecute it any time it does regardless of how often.