You can interact with legitimate financial institutions in the normal way under your public identity. You've just got to make sure you don't share that identity with your bitcoin counterparties, which will limit your ability to exchange bitcoin for things of value such as fiat currency.