Oh sure, you can sue for anything, and defending yourself is expensive. This is a problem in general, but not one worth worrying about in practice (will I get sued for writing this comment - I could!), but I'm a lot more interested in this part of your comment:

> and possibly even win, given that it's probably hard to back up a lot of these accusations in a way that would hold up in court

Assuming I didn't sign any explicit agreement (like as part of a settlement), if I say, "I did not find X to be a good employer because of Y", I do not believe it is necessary to prove Y in court in order to win a defamation suit.

But we're both guessing. I would say that your side would be easier to find evidence of, though, if you're right: do you know of, or can you find, examples of this sort of suit being brought and won?