Have you tried...
a) BM25 after some preprocessing (lemmatization etc.)
b) fastText / GloVe (possibly weighted by BM25)
The results can be surprisingly good. Often no need to bother with big language models or GPUs.
Re:Practical Deep Learning for Coders 2022
by sriram_malhar on Saturday July 23, 2022 @12:52PM
(#32205074)
As far as I understand, BM25 is not for fuzzy searches. For a bit more context, the search terms are in the English script, but the words are basically the closest-sounding transcriptions of sounds in various Indian languages. Different people may render the same word differently in an English transcription. But there's enough crowd-sourced data to account for the ways in which words can vary.
For the same reason, GloVe is of not much use to me.
OK, I understand. That is a different problem indeed. Although I am not sure that BERT would be of much more help there either, unless you have quite a large training corpus. The simplest/cheapest approach might be some kind of transcription normalization. As you say, what is introducing ambiguity here is the act of transcribing these sounds into the English alphabet. There’s not a single source language/alphabet? Also, I am not sure if fastText/GloVe really would not pick up the semantic similarity in spite of differing transcriptions as long as you have a large enough training corpus. I’d experiment with the settings here (bag of words/skipgram, min/max lengths etc.).
Re:Practical Deep Learning for Coders 2022
by devnonymous on Friday July 22, 2022 @03:53AM
(#32189626)
If the volume of data fits on a mobile phone for it to be offline, perhaps you don't need deep learning?
Re:Practical Deep Learning for Coders 2022
by sriram_malhar on Saturday July 23, 2022 @01:04PM
(#32205207)
Perhaps. What alternative would you suggest? The search terms are fuzzy, and there are too many variants (not exactly misspellings) for me to encode them explicitly. So I thought I'd rather learn from a crowd-sourced corpus.
In my case, I don't want the model to be general. I can afford for it to be like a database index, tailored to that data.
I have a search problem of my own and I have had a hard time applying what I have learnt (including the coursera DL specialization). The chief characteristics are: (a) It is a fuzzy search of a corpus that is in a non-English language. (b) The search should be able to run on a mobile phone _offline_.
Is this possible? Can training be done elsewhere and transferred to TinyML or some such? What would be a good forum to go seeking answers?