You can't just hand wave that ambiguity away and pretend that only the OSI definition is understood and used. You have to accept the "open source" is an ambiguous term and you will need to use qualifiers or additional context if you want to avoid ambiguity when using that term.
Yet people really only know of Open Source as a common term to describe these projects.
Simply because the source is openly available to read, the licence can still say "you can read but no touchy touchy or compiling it".
While technically it doesn't stop you from actually doing this, in a court of law you'd be found to have violated the licence. So it wrecks the idea of businesses doing it, because usually the licence holder won't care unless someone is making money from it.
Free software lets you do whatever you want, but you gotta show your code. Big difference in practice, especially when it comes to running a business.
Worst case? You walk through use life using terms in non-standard, idiosyncratic ways, and people are constantly confused by your speech, or argue with you because you don't have a shared understanding of the world.