I asked the authors of the original SMILES paper and they didn't have a good answer. I wrote a parser for SMILES so I could tokenize that way but never followed up, and eventually SELFIES was announced.