It sounds like you are talking about credit cards and not bitcoin.
Compared to credit cards, bitcoin has extremely low fees and fast payments. The fraud protection for merchants is far superior compared to any credit card payment solution.
It seems pretty obvious that most people spreading this kind of bullshit have never actually tried to use bitcoin, and have instead built their knowledge based on forum posts by other uninformed people.
The other reason it matters is that if something is primarily used for illegal activities that'll really hold back the whole system: most businesses aren't going to offer a payment option which makes them look shady, people are going to hesitate to use have the software installed, running an exchange will be even riskier, etc.
For Bitcoin in particular, there's also the practical risk that doing something illegal in a public ledger with non-repudiation guarantees is not generally a good call.
I’m not saying it can’t be used but just that there aren’t many reasons other than speculation or an ideological stance. If I was, say, the corner coffee shop what would balance out the extra fees, delays, and lack of fraud protection?