Re:Algolia Acquires Search.io
by marcinzm on Thursday September 15, 2022 @08:23AM
(#32850671)
Aren't typos just a question of how you generate your vectors/embeddings? I'd be surprised if a transformer with a character level tokenizer trained on a representative source of data (ie: with typos) wouldn't be able to make sense of typos.
[ Parent
| on HN ]
Re:Algolia Acquires Search.io
by evrydayhustling on Thursday September 15, 2022 @08:31AM
(#32850781)
Can confirm. We use sentence-level transformer embeddings for (vector) search, clustering, and classification tasks. As an old school ML guy I've been amazed at how robust they are to typos, slang, punctuation, etc.
However, I'm sure there are still applications where you don't have access to a robust embedding for your domain but can apply other techniques to deal with that domain's noise.
Re:Algolia Acquires Search.io
by O__________O on Thursday September 15, 2022 @09:06AM
(#32851296)
Here is decent intro to sentence level transformers & embeddings:
Re:Algolia Acquires Search.io
by dustincoates on Thursday September 15, 2022 @08:35AM
(#32850836)
Yes, good point. I still believe that net-net you're going to get better results on typos with a keyword-based search, but I didn't mean to imply that vector searching won't handle typos at all.