News for hackers. Stuff that matters.
Re:GPT-3.5 Turbo fine-tuning and API updates by danielbln on Tuesday August 22, 2023 @03:15PM (#37227315)
If you do a lot of retrieval augmentation, then the 8x cost might still be cheaper than burning a lot of tokens on injected context.
Re:GPT-3.5 Turbo fine-tuning and API updates by BoorishBears on Tuesday August 22, 2023 @03:27PM (#37227438)
Fine tuning is not a replacement for retrieval augmentation. If you wanted to do that you'd be better off fine-tuning BERT to retrieve data more effectively (identify key points, smarter summarization, cross-encoder relevancy, etc)
Re:GPT-3.5 Turbo fine-tuning and API updates by danielbln on Tuesday August 22, 2023 @03:32PM (#37227490)
Could you elaborate? It may not be a replacement, but I can think of a few use cases where RAG could be avoided entirely by having the model be fine tuned on the right dataset.
Re:GPT-3.5 Turbo fine-tuning and API updates by minimaxir on Tuesday August 22, 2023 @03:37PM (#37227551)
There is a risk of (greater) hallucination if there aren't explicit sample inputs provided by RAG.
Re:GPT-3.5 Turbo fine-tuning and API updates by huac on Tuesday August 22, 2023 @04:03PM (#37227834)
Finetuning teaches skills. RAG provides 'facts.'
Re:GPT-3.5 Turbo fine-tuning and API updates by lukev on Tuesday August 22, 2023 @04:12PM (#37227924)
Hallucination, but also the ability to cite a source.

A model using RAG can tell you why it answered a certain way, and cite chapter and verse from the underlying data. That's impossible just from model weights.

Re:GPT-3.5 Turbo fine-tuning and API updates by stavros on Tuesday August 22, 2023 @04:27PM (#37228107)
Speaking of RAG, does anyone know of a Python library that (hopefully) doesn't need a server, that can do embeddings, along with an example? I want to do RAG in Python, and I can wrangle the prompt to where I want it, but I have no idea how to basically search for relevant documents with embeddings.
Re:GPT-3.5 Turbo fine-tuning and API updates by lukev on Tuesday August 22, 2023 @04:39PM (#37228208)
https://www.trychroma.com

It works as a library, you can require the PIP package and get started immediately.

Re:GPT-3.5 Turbo fine-tuning and API updates by stavros on Tuesday August 22, 2023 @04:42PM (#37228237)
This looks great, thank you!
Re:GPT-3.5 Turbo fine-tuning and API updates by J_Shelby_J on Tuesday August 22, 2023 @08:25PM (#37230375)
Here is my implementation used for my discord and slack bots. The logic for rag is the Shelby_agent module.

RAG is straightforward but i improve it by generating extra keywords to help the search. I also add extra metadata filtering to return both technical documentation and more fluffy sales and marketing stuff. IMO, semantic search alone might not be the best, and I want to also add keyword search.

I think the thing most people will struggle with is prepping the documents. Chunking them to the best token count for your use case for example.

https://github.com/shelby-as-a/shelby-as-a-service

Re:GPT-3.5 Turbo fine-tuning and API updates by stavros on Tuesday August 22, 2023 @08:27PM (#37230393)
That's very helpful, thanks!
Re:GPT-3.5 Turbo fine-tuning and API updates by stavros on Tuesday August 22, 2023 @09:58PM (#37230996)
I know I left another comment, but I'm replying in a new one to increase the chance that you'll see this: It would probably be very useful to a lot of people if you (or someone) made a standalone library out of this. It looks like Shelby requires Pinecone, but using something like ChromaDB for local embeddings and providing a library that I can just import, add documents to, and then query for answers would enable all kinds of applications.

I'm currently trying to build a hobby virtual assistant, and it looks like RAG does need some nuance to implement correctly (from what you said). A library that does all of what you mentioned would take a lot of work out of it.

Re:GPT-3.5 Turbo fine-tuning and API updates by J_Shelby_J on Wednesday August 23, 2023 @01:32PM (#37239400)
That's actually something that I think is a next step.

a) run locally through a web ui b) run with a local vectorstore

From my perspective of someone using it as a bot that runs in a container using a cloud based vector store makes a lot of sense, but yeah that's really good feedback and not something I considered. Hrmmm. Ok let me get a look at some local vectorstores.

Re:GPT-3.5 Turbo fine-tuning and API updates by davedx on Wednesday August 23, 2023 @04:22AM (#37233237)
That doesn't make sense - if you're doing retrieval augmentation, then every prompt will be different depending on the query, right? So you still need to burn those tokens.
Re:GPT-3.5 Turbo fine-tuning and API updates by danielbln on Wednesday August 23, 2023 @04:37AM (#37233317)
You can persist the knowledge you would normally retrieve dynamically and inject into the context at runtime via finetuning directly into the model weights. This means you won't have to pull in a glob of additional data with every prompt, but only need to push the query through, which can be answered by the data persisted in the weights.

However, as others have pointed out, you lose the ability to reference sources reliably and the output is much more prone to hallucination.

Re:GPT-3.5 Turbo fine-tuning and API updates by davedx on Wednesday August 23, 2023 @04:52AM (#37233403)
Ahh understood, thank you.