Can you list a few I'm interested in checking them out

Slightly outdated article, but still relevant imo to show the different types.

https://medium.com/@nils_reimers/openai-gpt-3-text-embedding...

I've also used https://huggingface.co/flax-sentence-embeddings/all_datasets...

I am a fairly technical guy (check out my submissions) and I read your links and have no idea how to use these to make responses the way I can with OpenAI.

It says I can input a Source Sentence and compare it to other sentences. For example, how do I get it to reply to a question as if I am George from Seinfeld?

Embeddings are not for that. Embeddings take text and encode it into a high dimensional vector space. Similar texts will be closer together in the vector space.

The idea I was proposing was to use embeddings as a way to store and retrieve relevant "memories" so the AI could maintain coherence across time. I.e. whenever the user sends a message, we pull up the N most relevant memories (where relevance == closeness in the vector space) and include those in the prompt, so GPT3 can use the information when it forms its response.

I just implemented exactly this. In the corpus I put a few hundred papers I am interested in. Now I can ask a question, the search engine will find a few snippets and put them in the GPT-3 prompt.

Any good guides for embedding generation?

Yes this would be useful - does anyone have a crash course or something similar?

As I can't reply to the child - that makes sense it is for embeddings. So would GPT3 still need to be used combined with this then?

HN prevents users from responding to responses to their own comments without some delay to prevent flame wars -- just wait a few minutes next time, or click on the link to the comment directly and you'll be able to reply.

Yes you would still need GPT3 in this system. Right now, the incredibly simple system just wires gives GPT3 a window of the last 100 messages and has it output the next message to send.

    The following is an excerpt SMS conversation between two friends:

    Transcript:
    <splice in the last 100 messages here>

Then you can have GPT3 output what it believes the most likely next message is, and you send it. But this system means it loses context if a message is outside the window. So you can augment this system by creating an embedding of the last few messages of the conversation, and creating a prompt like:

    The following is an excerpt SMS conversation between two friends, and relevant past memories that are related to the current conversation:

    Relevant past memories:
    <splice in the N past messages with the most similar embedding to the most recent messages>

    Transcript:
    <splice in the last 100 messages>

So this gets you a kind of short term memory (the last 100 messages) and a long term memory (the embeddings).

> Relevant past memories: > <splice in the N past messages with the most similar embedding to the most recent messages>

This is a really good idea. Presumably you'd keep the memory per-person.

Oh, that makes a ton of sense. Thank you!

Thanks for the links, will check this out. It does seem compelling.

Here's a bunch being benchmarked

https://huggingface.co/spaces/mteb/leaderboard

You can call any of the huggingface models [1] using their API [2]. A few examples:

- https://huggingface.co/EleutherAI/gpt-j-6B

- https://huggingface.co/t5-base

- https://huggingface.co/facebook/opt-66b

- https://huggingface.co/bigscience/bloomz-3b

There are also other companies offering large models as a service:

- https://www.forefront.ai

- https://nlpcloud.com

- https://www.goose.ai

- https://cohere.ai/generate

[1] https://huggingface.co/models

[2] https://huggingface.co/inference-api

I wouldn't have suggested those models. Just use a semantically fine-tuned BERT.

> GPT-3 Embeddings by @OpenAI was announced this week. I was excited and tested them on 20 datasets. Sadly they are worse than open models that are 1000 x smaller

https://twitter.com/Nils_Reimers/status/1487014195568775173

Get models here: https://sbert.net/docs/pretrained_models.html

Could you name some you have in mind?

I wonder if there are any that can be used as a general chatbot way easily.