There are some language models like DeepMind RETRO that can make use of a 1TB text collection after the model is trained. The idea is to chunk up the text and make the blocks searchable with a neural embedding index. When you ask a question to the model, it first searches for relevant information and then adds it to the prompt. The result - you can get GPT-3 quality on a 25x smaller model. That means you could have your own GPT-3. Another advantage is that you can reference the sources and the generated text is more grounded.

An open source project could make a "search-engine-language-model" and in turn make this library much more accessible.

A recent article related to this: http://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html