Re:Pirate Library Mirror: Preserving 7TB of books (that are ...
by visarga on Monday July 04, 2022 @12:21AM
(#31973459)
There are some language models like DeepMind RETRO that can make use of a 1TB text collection after the model is trained. The idea is to chunk up the text and make the blocks searchable with a neural embedding index. When you ask a question to the model, it first searches for relevant information and then adds it to the prompt. The result - you can get GPT-3 quality on a 25x smaller model. That means you could have your own GPT-3. Another advantage is that you can reference the sources and the generated text is more grounded.
An open source project could make a "search-engine-language-model" and in turn make this library much more accessible.
A recent article related to this: http://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html
Re:Pirate Library Mirror: Preserving 7TB of books (that are ...
by Vetch on Monday July 04, 2022 @02:59AM
(#31974257)
The interesting aspect of GPT-3+ is their reasoning like behavior such as Chains of thought or "step by step" generation. There's no evidence RETRO as is has sufficient capacity for the actually interesting behaviors.
Re:Pirate Library Mirror: Preserving 7TB of books (that are ...
by visarga on Monday July 04, 2022 @03:25AM
(#31974407)
I agree, that's why I asked the authors the same question : https://twitter.com/visarga/status/1543826941052166144
A web server can run just fine on e.g. Raspberry Pi Zero W ($10), exposing any such content to any smartphone etc able to connect to it via WiFi (Kiwix sells preconfigured SD cards for their content, even). So, assuming that most people already have a phone or a laptop or even something like a Kindle, the only non-negligible cost here is storage. And a 1 Tb SD card can be had for under $150 right now.
So if anything, I think OP is overly conservative, given that their estimate was "$200 per person". Unless that counts the devices used to consume the content, and not just storage and the server.