Wow, this is absolutely deviously simple, and drop-dead obvious in hindsight. Unless this person is copying someone else without knowing it, this little blog post is the most important LLM paper to come out in months! Assuming I understood it properly, ofc.

Some random musings off the top of my head on quantitative analysis of tokenized text:

1. Could you enhance this by generating piecemeal summaries and comparing those to find chunks? Presumably it would decrease noise.

2. If we’re dealing with graphable data, can’t we just apply some old-school clustering algorithms to it? Say, KNN?

3. Idk if this is implied or not already these days, but can you filter out incidental words before tokenizing to get a cleaner analysis? I.e. take out the prepositions to better highlight the unique words for each chunk?