Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv...
by Kutsuya on Friday October 27, 2023 @06:12PM
(#38044674)
Thank you for sending this. It's still quite puzzling to me if it's actually possible or not. Maybe what I want to train is a style? But then again, it should also remember other important things related to the friend..
Re:Jina AI Launches First Open-Source 8K Text Embedding, Riv...
by sainez on Sunday October 29, 2023 @05:13PM
(#38062873)
Parent comment is on the right track. It sounds like you want to fine tune an llm to mimic the conversation style between you and your friend. Then you can use a general embedding model to implement RAG so that the application can "recall" pieces of your conversation.
What you are asking for sounds like fine tuning an existing LLM...where the data will be tokenized but the outcomes are different? There is a lot of writeups on how people have done it. You should especially follow some of the work on Huggingface. To replicate talking to your friend though, you will need a very large dataset to train off of I would think and its unclear to me if you can just fine-tune it or you would need to train a model from scratch. So a dataset with 10s of thousands of examples and then you need to train it on a GPU.
https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehe...