This doesn't need to be the case, though depending on context extra design elements (colour, descriptive text, …) may be needed to make the interpretation truly unambiguous.

Sometimes (advert stalking preferences being the key example) the confusion is very deliberately not dealt with, and furthermore on occasion extra steps are actively taken to ensure the confusion.