I understand that people want to look at this like a kind of slippery slope (if you can define one kind of restriction, why not define another and another - until you have closed source software by any other name) - but just because arbitrary restrictions mean something isn't colloquially "open" doesn't mean that certain specific restrictions might not still reasonably be compatible with something described as "open".
Certainly we should not accept a formalized definition of any language term like this; i.e. just because an OSI or other group wishes to claim the definition of open source doesn't mean we all need to use that definition. Specifically, let's not accept https://opensource.org/docs/osd as a kind of gospel truth. It's a fine initiative, but not necessarily the only possible one.
I'm not convinced that allowing a few restrictions would be a great idea, but then again, I'm also sure it would be dramatically different from the status quo, nor that is makes sense to no longer call such software open source.