News for hackers. Stuff that matters.

A comment on Show HN: LLaMA tokenizer that runs in browser.

Re:Show HN: LLaMA tokenizer that runs in browser by Hedepig on Wednesday June 14, 2023 @02:53AM (#36322354)
Somewhat tangimential, are there any open source attempts to compete with OpenAI's embeddings?

I know Word2Vec is a thing but I believe that is on a word by word basis, and doesn't capture the semantic meaning of whole sentences and paragraphs.

They charge so little for embeddings I secretly hope they do open source it. Because if for some reason it is stopped, any search functionality or the like that relies upon the API would cease to function

[ Parent | on HN ]
Re:Show HN: LLaMA tokenizer that runs in browser by dhruv_anand on Wednesday June 14, 2023 @04:05AM (#36322918)
https://news.ycombinator.com/item?id=36105660

Models under sentence-transformers are commonly used by people.

You can check this leaderboard, where OpenAI's embeddings are outperformed by open source ones: https://huggingface.co/spaces/mteb/leaderboard

[ Parent | on HN ]