Perhaps your decision to leave will be easier if you realize that if they needed to let you go, for whatever reason, they'd do it in a heartbeat. Not out of malice, and perhaps not without regret, but as a rational business decision.