Re:GPT3/DALL-E2 in Discord, chat like ChatGPT, generate imag...
by Workaccount2 on Thursday December 29, 2022 @10:05AM
(#34173294)
Sure, that is chatGPT in late 2022.
What about Open.ChatGPT in mid 2024?
Re:GPT3/DALL-E2 in Discord, chat like ChatGPT, generate imag...
by dpflan on Thursday December 29, 2022 @10:33AM
(#34173630)
Is there a "law of tokens" growth for LLMs, ala Moore's Law, but for LLM capabilities based upon token capacity?
Re:GPT3/DALL-E2 in Discord, chat like ChatGPT, generate imag...
by visarga on Thursday December 29, 2022 @12:55PM
(#34175583)
Complexity is quadratic in sequence length. For 512 tokens it is 262K, but for 4000 tokens it becomes 16M and goes OOM on a single GPU. We need about 100K-1M tokens to load whole books at once.
Since 2017 there have been hundreds of attempts to bring O(N^2) to O(N), but none of them replaced the vanilla attention yet in large models. They lose on accuracy. Maybe Flash attention has a shot (https://arxiv.org/abs/2205.14135).
I think we need LLMs capable of reading a whole book at once, about 100K tokens. I hope they can innovate the memory system of transformers a bit, there are ideas and papers, but they don't seem to get attention.