But what about the phrase "eliminate the ban on X." Then you're not banning something.

It's almost like you can't just look at the verbs alone and distill very useful meaning.