Re:Auto-Regressive Next-Token Predictors Are Universal Learners
by PaulHoule on Saturday September 23, 2023 @08:33AM
(#37622706)
I think the case can be overstated, or at least there are problems.
In the prehistory of transformers I was training character-based LSTMs and GRUs to write fake abstracts for PubMed abstracts of clinical case studies as a proxy for clinical notes.
This was before conversational models and prompts and one big problems was the system started out in the same state. Holistically the system has to decide if the patient has pancreatic cancer or athlete’s foot and a coherent story but really the system had to pick one of 26 letters to start with and then pick another character, assuming it spells good it is starting out in a constrained state space and on a knife edge between writing the same abstract over and over and writing gibberish based on the temperature.
At the time we thought putting in an initial state as an embedding would have helped, although for reading clinical reports we’d rather have a final state as an embedding.
Knowing what I know now we should have made up prompts (“Case study of a 23-year old man with a fractured tibia who presented at the emergency room:”) and stuck them in front of the abstract but of course that would have meant actually reading the 80,000 abstracts (40 person x days of work, our team could have done it in 2 weeks)
The thing is there is a gap between what is possible and what is practical. A good author has an end in mind and writes something and rewrites it and I have been in so many conversations with people speculating about the inference algorithm used by ChatGPT because so often it seems implausible you could really get good results token-at-time.
In the prehistory of transformers I was training character-based LSTMs and GRUs to write fake abstracts for PubMed abstracts of clinical case studies as a proxy for clinical notes.
This was before conversational models and prompts and one big problems was the system started out in the same state. Holistically the system has to decide if the patient has pancreatic cancer or athlete’s foot and a coherent story but really the system had to pick one of 26 letters to start with and then pick another character, assuming it spells good it is starting out in a constrained state space and on a knife edge between writing the same abstract over and over and writing gibberish based on the temperature.
At the time we thought putting in an initial state as an embedding would have helped, although for reading clinical reports we’d rather have a final state as an embedding.
Knowing what I know now we should have made up prompts (“Case study of a 23-year old man with a fractured tibia who presented at the emergency room:”) and stuck them in front of the abstract but of course that would have meant actually reading the 80,000 abstracts (40 person x days of work, our team could have done it in 2 weeks)
The thing is there is a gap between what is possible and what is practical. A good author has an end in mind and writes something and rewrites it and I have been in so many conversations with people speculating about the inference algorithm used by ChatGPT because so often it seems implausible you could really get good results token-at-time.