This would be a fun way to experiment with the effects of different tokenization strategies. For example, I've often wondered if tokenizing on phonemes would result in models that do better with creative writing exercises. It probably all gets lost in the wash when models reach terrible scale, but for small ones it seems it would potentially yield some benefit.