At the risk of being accused of a slippery slope, you could use similar logic to control speech of others too.

I think the real justification is part of prison is living a life of servitude where you basically have no control and everyone else makes the decision for you. America already more or less decided free speech was a danger worth accepting, and limiting prisoner communication won't put a dent on the collateral from that.