Re:Introducing ChatGPT and Whisper APIs
by vishal0123 on Wednesday March 01, 2023 @08:00PM
(#34990580)
https://arxiv.org/pdf/2001.08361.pdf. See the C_forward formula approxiamtion.
Thank you. Though it isn't quite clear to me whether the additive part is negligible?
From eq 2.2, additive part is usually in few 10s of millions. So, for N > 1B, approximation should be good but it doesn't work. For example, GPT3 inference flops is actually 3.4E+18 so the ratio is 19,000 not 2.
Re:Introducing ChatGPT and Whisper APIs
by vishal0123 on Thursday March 02, 2023 @05:12AM
(#34994028)
From the paper
> For contexts and models with d_model > n_ctx/12, the context-dependent computational cost per token is a relatively small fraction of the total compute.
For GPT3, n_ctx is 4096 and d_model is 12228 >> 4096/12.
Does someone have a source for this?
(By the way, it is unknown how many parameters GPT-3.5 has, the foundation model which powers finetuned models like ChatGPT and text-davinci-003. GPT-3 had 175 billion parameters, but per the Hoffmann et al Chinchilla paper it wasn't trained compute efficiently, i.e. it had too many parameters relative to its amount of training data. It seems likely that GPT-3.5 was trained on more data with fewer parameters, similar to Chinchilla. GPT-3: 175B parameters, 300B tokens; Chinchilla: 70B parameters, 1.4T tokens.)