News for hackers. Stuff that matters.
Re:Introducing GPTs by colordrops on Monday November 06, 2023 @02:51PM (#38167936)
I haven't been paying attention, why are embeddings not needed anymore?
Re:Introducing GPTs by sharemywin on Monday November 06, 2023 @03:11PM (#38168238)
Retrieval: augments the assistant with knowledge from outside our models, such as proprietary domain data, product information or documents provided by your users. This means you don’t need to compute and store embeddings for your documents, or implement chunking and search algorithms. The Assistants API optimizes what retrieval technique to use based on our experience building knowledge retrieval in ChatGPT.

The model then decides when to retrieve content based on the user Messages. The Assistants API automatically chooses between two retrieval techniques:

it either passes the file content in the prompt for short documents, or performs a vector search for longer documents Retrieval currently optimizes for quality by adding all relevant content to the context of model calls. We plan to introduce other retrieval strategies to enable developers to choose a different tradeoff between retrieval quality and model usage cost.

Re:Introducing GPTs by sjnair96 on Monday November 06, 2023 @03:38PM (#38168629)
Really cool to see the Assistants API's nuanced document retrieval methods. Do you index over the text besides chunking it up and generating embeddings? I'm curious about the indexing and the depth of analysis for longer docs, like assessing an author's tone chapter by chapter—vector search might have its limits there. Plus, the process to shape user queries into retrievable embeddings seems complex. Eager to hear more about these strategies, at least what you can spill!
Re:Introducing GPTs by riku_iki on Monday November 06, 2023 @04:12PM (#38169081)
> or performs a vector search for longer documents

so, clients upload all their docs to OpenAI database?..

Re:Introducing GPTs by emadabdulrahim on Monday November 06, 2023 @02:57PM (#38168018)
Re:Introducing GPTs by lazzlazzlazz on Monday November 06, 2023 @02:54PM (#38167967)
OP is incorrect. Embeddings are still needed since (1) context windows can't contain all data and (2) data memorization and continuous retraining is not yet viable.
Re:Introducing GPTs by zwily on Monday November 06, 2023 @03:16PM (#38168323)
But the common use case of using a vector DB to pull in augmentation appears to now be handled by the Assistants API. I haven't dug into the details yet but it appears you can upload files and the contents will be used (likely with some sort of vector searching happening behind the scenes).
Re:Introducing GPTs by nextworddev on Monday November 06, 2023 @03:13PM (#38168274)
"yet"
Re:Introducing GPTs by coding123 on Monday November 06, 2023 @03:18PM (#38168348)
It's also much slower. LLMs are generating text token at a time. That's not very good for search.

Pre-search tokenization however, probably a good fit for LLMs.

Re:Introducing GPTs by karmasimida on Monday November 06, 2023 @04:12PM (#38169086)
Embedding is poor man's context length increase. It essentially increases your context length but with loss.

There is a cost argument to make still, embedding-based approach will be cheaper and faster, but worse result than full text.

That being said, I don't see how those embedding startups compete with OpenAI, no one will be able to offer better embedding than OpenAI itself. It is hardly a convincing business.

The elephant in the room is the open source models aren't able to match up to OpenAI models, and it is qualitative, not quantitive.

Re:Introducing GPTs by estreeper on Monday November 06, 2023 @05:37PM (#38170120)
For embeddings specifically, there are multiple open source models that outperform OpenAI’s best model (text-embedding-ada-002) that you can see on the MTEB Leaderboard [1]

> embedding-based approach will be cheaper and faster, but worse result than full text

I’m not sure results would be worse, I think it depends on the extent to which the models are able to ignore irrelevant context, which is a problem [2]. Using retrieval can come closer to providing only relevant context.

1. https://huggingface.co/spaces/mteb/leaderboard

2. https://arxiv.org/abs/2302.00093

Re:Introducing GPTs by karmasimida on Tuesday November 07, 2023 @03:13AM (#38174395)
> on the MTEB Leaderboard

The point isn't about leaderboard. With increasing context length, the question is on whether we need embeddings or not. With longer context length, embeddings is no longer a necessity, and it lowers its value.

Re:Introducing GPTs by civilitty on Tuesday November 07, 2023 @06:06PM (#38184356)
For more trivial use cases, sure, but not for harder stuff like working with US law and precedent.

The US Code is on the order of tens of millions of tokens and I shudder to think how many billions of tokens make up all the judicial opinions that set or interpreted precedent.