If this dust is as radioactive as granite dust then its radioactivity is likely not a problem. (breathing in that dust may be a problem despite radioactivity itself not being an issue) Or maybe I am wrong and breathing in granite dust is more dangerous as it is more radioactive than some other stones?

It is 10 000 times more radioactive - then it likely is a problem.

The question is which one applies. Typical bananas are not dangerously radioactive despite being unusually radioactive for a food product.