Gangster-style yakuza rarely bother normal people. It’s not worth the trouble for either party. There are some industries with a lot of yakuza-owned and operated companies, but their main goal is to make or lose legitimate money, so unless you are aware of some tells you’d never know that they are yakuza.
Too bad japan isn't on the list. I would expect a high level of personal honesty there.
That said, there is still yakuza in japan. I wonder how that works.