First of all, thanks for your input and sharing your experience.
I think I'm a similar position to you for as so far that I don't really love math enough. My main motivator for doing it is to become more competent for other things I'm actually very interested in (algorithms, machine learning etc.). It's a means to an end and therefore a chore in some ways.
To put frame it in a more positive light: I'm confident that if I am able to grind through the first years of math, things will get much easier due to my much higher interest and a less workload during the latter years. There is some career/research advice that goes along the lines of "do what you think about when you shower". When I'm showering I often think about something relating to machine learning or similar.
As for medicine not being less mentally challenging, I don't think that holds true in my country. I know it's the case in most countries but here medicine is 6y and average study time is around 40h/week with only P/F grading.
I started a PhD and found that I just didn't love it enough. I think one's innate curiosity and drive regarding the investigation the subject matter needs to outweigh the undesirable aspects of the pursuit. I saw other people absolutely driven by their work, and they basically sailed through. The others who found it more of a chore often completed the program but then left the field.
It sounds like you and your program are not aligned.
The way you frame it, it almost seems like a no-brainer to switch. Perhaps that kind of work is just something you're more comfortable with. You can try these deep, introspective thought experiments, but I know it's hard to really know without actually doing it. The grass is always greener on the other side. Sometime it actually is.