Our use case was that we needed to consider information from thousands of pages of documents to answer customer questions. I let GPT decide where to look based on content tables of the documents (thankfully each section was not more than ~500 words) and only fetched information from the relevant sections to let GPT answer the user's question. GPT3.5 was fast enough to be usable but often inaccurate; it would ignore some information or get confused about simple mathematical comparisons in the user's questions like 'greater than versus less than'. GPT4 was good and potentially fast enough but the API throttling was too restrictive and it was not possible to feed it less information and still get a good, nuanced answer every time.

Actually the trick to get GPT to find the information based on table of contents worked really well in narrowing down the information for the second-stage summary (to answer the user's question) but our use case nevertheless required a lot of information (often from multiple sections) with a lot of grey areas and nuance and the limited amount of throughout provided by GPT4 didn't cut it.

It was frustrating that, aside from accuracy issues (which depended on the complexity of the question), it often seemed to work very well and fast with GPT3.5 but the exact same code did not meet usability expectations when we switched over to GPT4 due to slow speed (though the accuracy was awesome).

I only spent 1 month and a half on this project but management didn't have the patience to wait for Open AI to remove GPT4 throttling and because they couldn't see it working (at the speed they wanted), they had no reason to believe me that my solution was designed correctly... That's startups I guess.

Anyway I still think there is a lot of hope for this technology and I'm confident that the problem can and will be solved. It was just unfortunate timing for me. Being an early adopter isn't always a good thing.

From what you said your solution wasn't designed correctly?

You should definitely have been using BM25 and SBERT for something like this, you definitely should have been asking 3.5 for structured output and doing any math yourself.

If these were answers from a fixed set of documents there's also a ton of pre-processing you should have been doing.

-

I recently helped a friend with a problem of identifying a certain form of language in EDGAR sourced contracts. He had someone who tried feeding in entire documents into LLMs to find these and the result was some 20 minute search time.

I took a few minutes sitting with him to come up with a demo that used synthetic data and SBERT to process documents in less than 30 seconds. 99% of that was stuff you could do 2 years ago, the part the LLM helped with was creating buckets of synthetic data quickly, and even that could be done procedurally if you had more time.

>> you definitely should have been asking 3.5 for structured output and doing any math yourself.

I suspect that this solution would have required a lot more time given my lack of experience in these technologies and there are a lot of unknowns for that particular use case. For example, a user could ask to compare legalese from 30 different providers to find all those which met a certain conditions; even if you can narrow down the input to just a few sentences per provider, you still need to feed it 30 times that amount or else the AI has no way to compare all the providers.

With my solution, GPT only saw a tiny highly relevant fraction of all the documents. It's possible that the solutions you mentioned would have helped cut our input size further but it's not clear by how much given the complexity of the questions and the data.

Start your own company then, that's a product people would use.

(I'm making my own and agree with you, it works)

Why didn't you go the retrieval augmented generation route with a vector store?

That was going to be the next stage, but this would have been a more complicated solution with additional challenges but definitely a lot of code could have been re-used as it would just have been a matter of substituting the database lookup portion.

There are unknowns with the vector store solution because it doesn't suffice to just fetch a few relevant sentences in arbitrary order; we had to fetch every relevant piece of information in appropriately-sized chunks (some of which had to be multiple lines, some of which required the section heading for context) to formulate a correct answer. Sometimes there was something mentioned in a different section of the text which changed the outcome. Going down the vector database route would have taken longer and involved additional learning and it's not clear that we could have reduced the input size by doing that. I still think it was a good decision to start with a regular database first given that all information mapped neatly under the headings in our table of contents and that each section was relatively short.

>It was almost there; it was almost good enough such that we could almost ship

I feel like that a lot but we've been shipping for years.

What do you mean you lost your job over it? All because GPT4 was too slow for your project?

Yes. This is the reality in many startups. I should probably have refused to work on this project and suggested some other task; that's what my colleague did; he only worked on the front end part. I took a risk, I misjudged the tool and failed.

I wasn't even upset (I got a decent severance), I'd been let go over far more trivial things when I was a junior years ago... Like when the boss fired me the first monday after he met my girlfriend at the office christmas party because he thought she was out of my league. That's why I don't bring her (now wife) to any office events nowadays.