To add to this point, bitcoin transactions are not just public, they're permanently verifiable.

By making a transaction with an address you're not just telling anybody who knows you own that address that you've made a transaction; you've allowed anyone who later discovers you owned that address to prove you made that transaction, for the rest of history.