It's been a while since I was at a company that had that clause in it, but if I was in that situation in the future then at the minimum I'd want to ensure that I could still work on open source projects on my own. I was wondering what is the best way to phrase this with a potential employer (esp. if you are out of work, and desperately need that job)?

I was thinking along the lines of "Hey, I periodically do volunteer work for various charities, and I think this clause could hurt them. Could we add in that work not directly assigned to me and is related to a non-profit / charity is excluded from this clause?"

That way, if anything comes up, you could turn over copyright to the FSF or similar organization and you would be covered.