Haven't followed up on all the comments in it but speculates on why chain of thought improves when training on code.
This is a thing that's already fairly well known
Thought, as a cognitive process, can bridge the gap between these two realms, enabling individuals to move back and forth between rational and irrational modes of thinking, depending on the context and objectives at hand.
With data, unstructured text could be considered "irrational" and structured text (like code or a column in a database) could be considered "rational".
x = True
y = False
if y:
x = y
else:
x = not x
print(y)
The hypothesis is that pre-training on this task will force the model to learn a lot of circuits that deal purely with in-context, causal reasoning.
https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a0...
Yes, a lot of tokens are just for code.
Edit: Here as raw link for the poor mobile devices:
https://gist.githubusercontent.com/s-macke/ae83f6afb89794350...
There’s something poetic about ULL being a token, but NULL not being one.
If you want to look at mappings for individual tokens, sure, but if you actually want to tokenize text that contains more than 1 token, the process is very non trivial. I've been writing my own JavaScript LLaMA tokenizer for several days now (just the encode and decode functions, not training). I hope to release it this weekend. It's currently 400 lines of code + data.
Input string: " grabbed"
Tokenize that with the greedy algorithm, you get [17229, 2580] == [" grab", "bed"]
Tokenize that with actual LLaMA tokenizer, you get [2646, 1327, 287] == [" gra", "bb", "ed"]
Note that the correct tokenizer represents this string with 3 tokens, even though it would be more efficient to represent this string with 2 tokens (yes, those 2 tokens exist in the vocabulary).
LLaMA uses SentencePiece Byte-Pair Encoding for tokenization, and it has many weird quirks like this.
You don't have to use tiktoken if you aren't actually tokenizing things. The token lists are just text files that consist of the characters base64 encoded followed by the numeric ID. If you want to explore the list you can just download them and decode them yourself.
I find that sorting tokens by length makes it a bit easier to get a feel for what's in there.
GPT-4 has a token vocabulary about twice the size of GPT-3.5.
The most interesting thing to me about the GPT-4 token list is how dominated it is by non-natural languages. It's not as simple as English tokenizing more efficiently than Spanish because of frequency. The most common language after English is code. A huge number of tokens are allocated to even not very common things found in code, like "ValidateAntiForgeryToken" or "_InternalArray". From eyeballing the list I'd guess about half the tokens seem to be from source code.
My guess is that it's not a coincidence that GPT-4 both trained on a lot of code and is also the leading model. I suspect we're going to discover at some point, or maybe OpenAI already did, that training on code isn't just a neat trick to get an LLM that can knock out scripts. Maybe it's fundamentally useful to train the model to reason logically and think clearly. The highly structured and unambiguous yet also complex thought that code represents is probably a great way for the model to really level up its thought processes. Ilya Sutskever mentioned in an interview that one of the bottlenecks they face on training something smarter than GPT-4 is getting access to "more complex thought". If this is true then it's possible the Microsoft collaboration will prove an enduring competitive advantage for OpenAI, as it gives them access to the bulk GitHub corpus which is probably quite hard to scrape otherwise.