Re:Writing a full-text search engine using Bloom filters (2013)
by umvi on Wednesday June 10, 2020 @12:08AM
(#23473829)
Doesn't seem like it would be that hard... Couldn't the server just run grep if a user searches? Grep could scan every static file in milliseconds
[ Parent
| on HN ]
Re:Writing a full-text search engine using Bloom filters (2013)
by stevekemp on Wednesday June 10, 2020 @12:56AM
(#23474077)
(If you go down this route you might prefer ag/the silver surfer/similar, as being faster than grep.)
I've actually reported a couple of security bugs in servers in the past, which were caused by people running grep for searches.
Imagine you have a tree of web-content at /var/www, when a user searches for "foo" you run "grep -r -h foo /var/www", then you parse out the resulting filenames.
Now imagine what happens if your user searches for "foo /tmp; rm -rf /". Whenever you pass user-input to a command-line application you need to be damn sure you know what you're doing - if the shell is involved in parsing your command you'll have problems.
Re:Writing a full-text search engine using Bloom filters (2013)
by kragen on Wednesday June 10, 2020 @03:17AM
(#23474763)
This is a good point. Even if you don't have a shell injection vulnerability, you may end up with a path traversal vulnerability. In this particular case I think you're probably pretty safe as long as you're careful to only pass a single argument to the grep process — but it depends on how exactly grep parses its arguments. If there's a way to sneak ../ in there, you might get hurt.
Bleh. I can see why people install WAFs.
Re:Writing a full-text search engine using Bloom filters (2013)
by stavros on Wednesday June 10, 2020 @07:25AM
(#23475894)
I've lately become allergic to things that I can only partially reason about. Sure, I'm probably going to be fine, but I can't guarantee I'll be fine, and the blackhats are always more knowledgeable than me about obscure bits of shell/Unix behavior, so I just shy away and use a less free-form approach.
Re:Writing a full-text search engine using Bloom filters (2013)
by djur on Wednesday June 10, 2020 @12:21AM
(#23473885)
If you're running grep it isn't a static site anymore.
Re:Writing a full-text search engine using Bloom filters (2013)
by Nyubis on Wednesday June 10, 2020 @02:27AM
(#23474529)
The big difference here is that the bloom filter implementation runs entirely client-side. It saves a roudtrip per search.
Re:Writing a full-text search engine using Bloom filters (2013)
by bule0bird on Wednesday June 10, 2020 @12:41AM
(#23473986)
The indexing task can be done during static site generation and results shipped as resources, but the query part (search string parsing, ranking, snippet generation, etc) needs some code running locally (browser, js).
We've used Minimal Perfect Hash Function with bloom filter to reduce the mem of a service from ~27G to ~8G, as the data was static. Probably something like that can be used here too, to make indices small.