Money on the other hand can do much more good not sitting in the bank account of someone who has more of it than they could ever spend. So keeping it in there for the purpose of 'scoring points' is seen by many, myself included, as obscene.
Also, I disagree that collecting money is the only hobby that would receive this criticism. Personally I would also find it obscene to collect insulin, clean water, and residential housing to name a few.
It especially punishes the middle class because the money in their wallets would buy less proportionally.
If anything ideally proportionally less money should be available for chasing after consumer goods.
Besieds his money doesn't sit it in the bank. He has it in short term tresury bonds. Which means he basically lent it all to the government. If this money is not used to help the poor then blame the government because it controlls Warren Buffett money until it gives it back to him.