You're missing context. In the stock market, you get paid for believing/knowing things others don't. Ray is calling that "smart"ness.

If you know better than anyone else, there's a fortune to be made on Wall Street. A problem is that the way he uses "smart" is not in line with how non-financial people think, so when these phrases/anecdotes escape their context it sounds... uh... the way it does.

Sure, there might be a better way to phrase what he's saying. But he's not wrong.

(Separately, there's a reason why tweets with 500 words of qualifiers and "it depends" and "YMMV" don't catch on. Brevity counts, especially when reminding people that they don't/can't know everything compared to The Market.)