I did this too for about 5 years, except I took 2 weeks off each quarter. I learned Scala, programming language theory, did papers on finance and logic, etc. The job I have now is based on some of the open source work I did during that period.

When I took time off to work on interesting stuff I found I was more practical and sensible in my main job. When I designed solutions I was very business focused. I didn't need to use the latest tech or do novel work, since I could do that in my own time. The day off each week kept me fresh and interested without needing to get that from my main job.