Legality matters for two reasons: one is simply that the huge predictions about market valuations and transformative social effects require more volume than people buying drugs online.

The other reason it matters is that if something is primarily used for illegal activities that'll really hold back the whole system: most businesses aren't going to offer a payment option which makes them look shady, people are going to hesitate to use have the software installed, running an exchange will be even riskier, etc.

For Bitcoin in particular, there's also the practical risk that doing something illegal in a public ledger with non-repudiation guarantees is not generally a good call.