I have attempted a couple times to re-learn and the result is always the same. It goes beyond anxiety. If I try to read so much as a three digit number, my brain turns to mush and things start to get blurry. I have trouble reading ticket numbers on a daily basis, and struggle to help my grade school kids with their homework. I _can_ do formulae and calculations when I really hunker down, but it takes me considerable time, and I'm often wrong.
I did a ee and cs degree in college and I hated the eng part of it. It was because I hated the exams which was mostly rote stuff. When I finished college though and started working in a job that had no linkage to ee I one day just picked up courses (which were just arbitrary web 1.0 sites by random folks) and started going through the (signal processing/kalman filtering/convolution) problems and implementing them in software which was a lot of fun to see in action.
Reason for that long winded one was, I was thinking in your case, would the anxiety still be there as now you (might) have a reason to develop passion for the math than back in school where it was forced without a reason?