While I encourage you in your projet, there are the usual aspects about IP. For example how can your second employer be sure that what you work on with them, will not be claimed as intellectual property of your primary employer?

Not exactly the best example but look at the controversy generated by Carmak going to Facebook.