Tell that to every human on HN and see if they start wearing their nighties and undergarments, and bikinis to their silicon valley workplaces.
It's not a free-for-all, so don't pretend that cultural norms don't exist.
Jeez, this "I'll take what I want and fuck ya'll" American culture is sickening.
And there's no way to decide what's "permitted" use of a piece of apparel anyways. Cultural significance changes within cultures, and individuals within it are not uniformly sensitive to informal use. What would you do to iconoclasts? They're culturally appropriating too. Are they allowed to introduce lighthearted use?
The "cultural appropriation" idea is the worst form of ultra-conservative hand wringing dressed up in progressive language. No actual harm is done by "misusing" clothes. This is just an excuse made up by moral busy bodies to go on yet another obnoxious crusade.
This is where the problem is. You misunderstand what I'm saying. Nobody is stopping you from wearing whatever you want.
When people call out cultural appropriation, they aren't trying to ban something; merely point out a faux pas.
As a reminder, all of this is about a pretend problem that produces no damage and has no victims.