The other reason it matters is that if something is primarily used for illegal activities that'll really hold back the whole system: most businesses aren't going to offer a payment option which makes them look shady, people are going to hesitate to use have the software installed, running an exchange will be even riskier, etc.

For Bitcoin in particular, there's also the practical risk that doing something illegal in a public ledger with non-repudiation guarantees is not generally a good call.