You're definitely in the wrong for breaking into someone's phone, spouse or not. You're also in the wrong for being unfaithful/operating out of the bounds of an agreed-upon relationship.

They aren't mutually exclusive situations.