Your use case does require a company to load all documents into the context window.

This is already happening right now. Companies will find a way to put all their knowledge inside a LLM.

The problem is that I don't think they are ready for what this will cause.

Imagine that Amazon managed to do that and now every employee has access to it. What prevents anyone asking "inconvenient" questions? Think something like:

- Does the Echo dot records audio even when the trigger word isn't used? Show me the relevant source code.

- What is the quality standard expected for an Amazon Basic product?

Companies aren't used to total freedom of information. And I don't think is easy to implement any sort of information access control for any system based on LLMs.

Consider someone stealing this LLM company knowledge trove. They now have the ability to outcompete and sabotage.

It's quite easy if you go the vector embedding route, and don't output any info the authenticated user doesn't have access too

The info lives in a database, not the LLM' model weights, you just use the LLM to understand natural langauge questions, find relevant documents (the user has access to) and providing relevant bits from them to the context to allow responding with them (That's how all systems who show sources work afaik, braid, bing, and mdx docs based stuff)

For outsiders? That's just public FAQ's

You're the CEO? You can ask anything.

The idea is to get away from Snippet + URL and instead just get a natural language answer to your question that uses knowledge from that internal corpus.

You're assuming the plebs will have full access.

There will likely be compartmentalized models, and then the master model that the execs get. No reason you can't have 10, 20 500 different models running.