Curious cat - had you considered using Antlr4 and a Python visitor/listener? (Were you aware of antlr?). Depending what you're trying to do with a regex tokenizer, it might be suitable.

It was more just for fun than anything else. I was more interested in the recursive descent parser so the tokenizer was just a step along the way.

I've played with Antlr but never really vibed with it to be honest (and it was years ago)