The real danger probably isn't that they'll try to enforce it per se, but that they could make you "radioactive" by the threat of it, or an enforcement action, even if ultimately unsuccessful. Hopefully no or few potential future employees will ask about them, and if this happened in California and you're still they (it gets more complicated if you're only now in the state), any non-compete provisions are unenforceable as such.