You may have fixed PROGRAMMER - MAN + WOMAN = HOMEMAKER, and still get SOLDIER - AMERICAN + ARAB = TERRORIST, the list is endless
The baked-in assumption that Arabs or Muslims are terrorists, or that terrorists are Arabs or Muslims, is something that the de-biasing process in ConceptNet Numberbatch (which I make) attempts to mitigate at the same time as gender and racial bias. And of course there is much more to do.
It's a fascinating and productive field of research. Why did you have such a negative initial reaction to it?
Ultimately the Model would need to detect & learn the contexts by itself.
And more to the point; I see you are publishing data sets after correcting them for fairness. I think the unfair results can sometimes be more useful. I once prototyped a short-story tagging & recommendation system based on word2vec and I would not be surprised if raw vectors gave better results. People's taste in litterature (especially romantic) are very dependant on gender stereotypes.
If you read the paper, you'll see that de-biasing is not based on assumptions, it is also based on data. Bolukbasi ran a pretty substantial crowdsourced survey to find the comparisons that word vectors make that people consider inappropriate. Don't make hollow demands for proof when you're not even aware of what's already been shown.
Crowdsourcing comes with its own set of biases, of course, and we may want to revisit this data sometime, but so far this is a pretty reasonable proxy for whether an ML model will cause problems when it is deployed. And it's much better than nothing, the option that I'm struggling to understand why you prefer.