I had a similar thing happen, where I had this insane aspiration of dual majoring in computer science and engineering and music. After doing marching band my first semester, which was 6+ hours of work a week ignoring games, and seeing how hard the music majors were working, I realized how difficult that'd be.
I still do the occasional ensemble, but the time you need to commit to those courses are insane.
After three semesters of majoring in music, I made things easier for myself as a student and switched to computer science, with a hefty dose of math on the side. (I also thought that might be better for employment prospects; I think I was right.)
I've continued to pursue music as well, but wow, those classes are intense.