Re:Books in .txt format for AI training purposes
by minimaxir on Sunday October 25, 2020 @03:33PM
(#24889190)
AI models suffer massive diminishing returns on both model size and amount of input text, which fortunately gives fresh models a chance to compete (the benefits of GPT-2/Transformer architectures is that they are capable of scaling indefinitely, but they still have diminishing returns)
[ Parent
| on HN ]
Re:Books in .txt format for AI training purposes
by Voloskaya on Sunday October 25, 2020 @05:16PM
(#24889956)
I completly agree, it's a diminishing return but a return nonetheles, so to have a "world class GPT" model with 10% of the data, and 10% (optimistically) of the compute that OpenAI used, you will need some very very impressive innovation elsewhere.