If a gun's loudness sometimes acts as a deterrent, then the availability of this "piece of metal" will reduce/eliminate this deterrent, and consequently increase the probability of the weapon's being used.
The same can't be said for ear protection.
"A 30-decibel reduction in theory means an AR-15 rifle would have a noise equivalent of 132 decibels. That is considered equivalent to a gunshot or a jackhammer." Are firearms with a silencer 'quiet'? - The Washington Post
What do you specifically mean by "proven to make a difference"? Suppressors are known for being used by intelligence services -- presumably they're not using them to protect their agents' hearing.
It should be blindingly obvious that silencers/suppressors have a use beyond "hearing safety". Namely, that they allow someone to shoot someone else without attracting as much attention.
Are you claiming otherwise?
The point is that given the availability of effective ear protection, there is no legitimate argument for civilian use of suppressors. There is no plausible argument that their existence would decrease the risk of a gun's being used illegally, and there are great reasons to believe that in some instances it would increase this risk, and/or make it more difficult to catch those who have done so.
Long range flamethrowers are illegal, but could be said to have a "legitimate use" for clearing brush. When there is no good argument for a thing to exist, and there are good safety-based arguments for it to not exist, then in the name of safety we should not have said thing.