Maybe find a team you think you can really impact with your PM+Dev skills, deliver an impactful project, then use that as leverage to get promoted within this team and/or a fleet of teams. I like what you have to say r.e. learning and enjoyment, but don't sell yourself short. You're probably a lot more valuable to the group/division than you realize.
Biggest downside is economic; can't negotiate new pay when changing teams by policy and you've gotta build a new case for promos/etc. People who have changed companies 2-3 times are likely making significantly more money than I am. I try to keep it in mind that I'm paying for quality of life by staying, so I need to get my money's worth or it's not worth it.