It absolutely makes sense. If you're not doing math or writing code, then you're compiling reports and ticking boxes in checklists. Get out of infosec, and start building things.

You can jump to networking, systems automation, data science, product development... Anything else. If you return to security later, you'll be better equipped. If you pick up some theory whilst developing your chops, then so much the better.

Good luck!