If I understand your question right, this is one of the reasons BPE is nice and the parent liked it. For any character sequence, provided the characters are in the alphabet used to create the BPE vocab, there are no unknown words/sequences. One downside of some previous tokenization methods is you could have unknown/UNK tokens, EG dictionary based methods.
In our paper with bytes, we also avoid the UNK issue, since we can have an embedding for every possible byte, since itβs not that many (and for sequences of bytes we use hash embedding, although we did test n-gram lookups for the top K frequent byte n-grams in the training data).
Did you guys try using an RNN or some other kind of DNN to encode the patches?
Interesting. I would have thought one of those "minimum viable" RNNs (like https://arxiv.org/abs/2410.01201) would have been ideal for this. I might tinker a bit with this :-)
The real-world for humans has them creating and using non-dictionary words to communicate daily. A good example is "notify", defined in the dictionary. "notifier", which is not and is used to describe "a means to notify someone". The code to send an email notification is an "email notifier", then there is text message, voice call, call center call back notifiers ....
All industries and organizations have jargon, custom defined words not found in a dictionary and use non distinctive acronyms.
How would a ML output be useful if it cannot handle real world commutation and only lab based sanitization of in-dictionary only responses?