The other thing I found was that I was more productive when I was working fulltime, than when I had more time to work on my own project. I worked better when I had less time, since I was more focused, would only work on the most important stuff, and only had a few hours to get something done.

You may miss this, but hey, go for it anyway. If you leave on a good note, then there is always a chance of going back.