If we don't want the idea to be gradually chipped away until it's meaningless then yes, we need to draw a line in the sand and say that open source can have these restrictions but not those restrictions. And thankfully we already have that: the OSI's definition, Debian's DFSG, and the FSF's Free Software Decision are all substantially equivalent and represent a clear consensus on what is and isn't open-source; one that a lot of care and thought have already gone into.