Re:DuckDuckGo Regex Search
by unhappyhippie on Tuesday July 08, 2014 @12:57PM
(#8005464)
Maybe I am missing out potential applications but I can't understand why I would like to search the entire web with a regex pattern? Find all strings that could be SSNs?
For 580$, you can still do it:
http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-bil...
Wake me when someone does this properly. It's only been done on a small scale or with very limited precomputed expressions, to my knowledge. Not many people would need it, but for those people it'd be insanely useful. But it's also insanely computationally hard - which means it'd be a really interesting technical achievement! There are no general-purpose reverse indexes that I know of that accelerate that as easily as keywords, but there are some data structures that might help a bit, although I can't think of practical ways to deploy them over arbitrary regexps specified at runtime! Plus some sanity heuristics and limits, of course, as regexps can undergo combinatorial explosion and some fun unexpected worse-case performance.