I agree and I would also check the contract for an "Entire Agreement" clause which basically says that side agreements, whether verbal or written, are not considered part of the agreement.
I would agree with also checking with the employer. Any time you want to utilize open source in a project you should see what the employers policies are on this.
You will also need to be careful about contributions back to your open source stuff. This is one of the areas that open source foundations have done really well. The dojo and Apache foundations as an example have you fill out a contributors agreement to ensure the IP of any code that is submitted to them is "clean".
Also it might happen that while integrating it, improvements to the open source code come to mind. Then it should be clear from the start if it would be OK to feed improvements created during employed time back into the open source project.