Lunr.js [1] used to use a trie data structure and memory usage was a concern. The problem is that although the data structure compresses prefixes, it does nothing for suffixes. Depending on the corpus this can lead to a lot of duplication.
Switching to a trie-like structure that compresses prefixes and suffixes can lead to significant savings. Building the structure can be a bit more burdensome, so there is a trade off there. There is a paper describing the approach [2] and, if you're interested, my JavaScript implementation [3][4].
[2] http://www.aclweb.org/anthology/J00-1002.pdf
[3] https://github.com/olivernn/lunr.js/blob/master/lib/token_se...
[4] https://github.com/olivernn/lunr.js/blob/master/lib/token_se...
Cheers! Thanks for sharing. EDIT: I like the site design too!
Yeah, the memory is the only downside of Wade. I'm still looking into ways to optimize the trie used while still keeping the size around 1-2kb.
The stop words that are being removed can be configured via `Wade.config.stopWords`, which is an array of stop words that Wade will remove.
Also, if the library removes stop words, does that mean its limited in which languages it supports? I'm assuming only English stop words are filtered
Not being critical here, just genuinely curious and don't have time to look through the code ATM