News for hackers. Stuff that matters.
Re:SeenBefore: A search engine for what you have seen before by crntaylor on Monday September 24, 2012 @09:30AM (#4564527)
Obvious point to raise: the reason people regularly delete their browser history is because they watch porn without turning on private browsing. How do you propose to deal with this?

You'd need to provide at least the ability to selectively delete portions of the history. But you can selectively delete portions of your browser history too, and people don't - because it would be too easy to miss something. Instead, they just nuke the whole thing. How is your tool different?

Re:SeenBefore: A search engine for what you have seen before by rickdale on Monday September 24, 2012 @09:48AM (#4564622)
I take advantage of my browsers history with porn. When I was in college the first bash script I wrote was to open a movie in my porn collection that I hadn't watched in the longest time. This was great. But now with streaming porn sites I don't have a huge collection and I often watch scenes that I can't seem to find later. There is a lot of porn out there.

Sure people clear their browser history because their embarrassed by their porn obsession, but I think this tool could be very useful for pornaholics too.

Re:SeenBefore: A search engine for what you have seen before by beaumartinez on Monday September 24, 2012 @12:20PM (#4565389)
You could do something similar today—just hook into your browser's history API
Re:SeenBefore: A search engine for what you have seen before by vinnyglennon on Monday September 24, 2012 @10:14AM (#4564757)
Vinny Glennon, One of the founders here. Thanks very much for the up votes. The Chrome extension does not work in private browsing. I have a set of porn sites(1.7 million stored in redis) that I check if incoming links are a member of. You can selectively block sites ( https://www.seenbefore.com/blacklist_items).
Re:SeenBefore: A search engine for what you have seen before by andrewthornton on Monday September 24, 2012 @10:51AM (#4564944)
Where did you get your list from, and can you share it?
Re:SeenBefore: A search engine for what you have seen before by user24 on Monday September 24, 2012 @10:54AM (#4564958)
For research purposes.
Re:SeenBefore: A search engine for what you have seen before by jparishy on Monday September 24, 2012 @11:27PM (#4568412)
For science.
Re:SeenBefore: A search engine for what you have seen before by krassif on Thursday September 27, 2012 @04:50PM (#4582817)
Re:SeenBefore: A search engine for what you have seen before by gingerjoos on Monday September 24, 2012 @12:02PM (#4565268)
Do you store the entire set of 1.7 million entries in redis? Or is redis an index to data stored elsewhere, in a relational DB perhaps?

I was under the impression that redis wouldn't be all that useful to store a lot of data. Would be great if something as quick as redis could work with large data sets.

Re:SeenBefore: A search engine for what you have seen before by vinnyglennon on Monday September 24, 2012 @01:07PM (#4565718)
Storing a list of 1.7 million strings for us takes 70mb stored in memory. Testing for membership is an O(1) op. Very happy with it. We use mongo as a dumb data store as well as a bunch of other infrastructure tools, like http://circleci.com we could have only dreamt of years ago.
Re:SeenBefore: A search engine for what you have seen before by genwin on Monday September 24, 2012 @10:46PM (#4568270)
> Testing for membership is an O(1) op

Curious how. O(1) an array index lookup, not a string lookup, I thought.

Re:SeenBefore: A search engine for what you have seen before by milkshakes on Tuesday September 25, 2012 @12:19AM (#4568544)
Re:SeenBefore: A search engine for what you have seen before by genwin on Tuesday September 25, 2012 @12:33AM (#4568584)
Again just curious, but I'd still like to know how. Someone there asks the mod how it could be O(1), the mod replies it's a "hash table lookup". But Wikipedia at http://en.wikipedia.org/wiki/Big_O_notation suggests that such lookup is no faster than O(log log n). I think the redis info is incorrect.

O(1) implies that the location of the member in the list is already known, with no search required. I don't see how that could be the case when it's a key lookup. The key could be anywhere in the list, even if the list is sorted. They key would have to be searched for, it seems.

Re:SeenBefore: A search engine for what you have seen before by milkshakes on Tuesday September 25, 2012 @01:08AM (#4568694)
O(1) means constant time. Redis sets are interesting, because there are a few possible implementations under the hood, but the typical case winds up implementing it as a hash table. Hash tables have constant time lookup.

Think of it this way (this isn't literally what happens, but it's close).

1) Take the url you're looking for. Run it through a hash function. This takes an (amortized) constant amount of time.

2) Now you have an index to check. (the return value from the hash function). So index into your actual table, and check to see what's stored there. If there's a value stored there, then the url is a member of the set. This also takes a constant amount of time.

Does this help?

Re:SeenBefore: A search engine for what you have seen before by genwin on Tuesday September 25, 2012 @01:23AM (#4568741)
> So index into your actual table, and check to see what's stored there

But that check isn't a constant time lookup. The lookup time can vary. (Analogously, a lookup in a phone book can vary in time; we can't necessarily go to the exact spot the first time.) So the total time for both steps must vary as well. I think.

Re:SeenBefore: A search engine for what you have seen before by milkshakes on Tuesday September 25, 2012 @01:29AM (#4568765)
I think where you're getting confused is that you're conceptualizing this like a search problem, where you compare values and inspect each member to see if it matches the target.

That's not what's going on here. Instead, you use the value as in input into a function that tells you where to look for it, then you look, to see if it's there.

If it's not there, it won't be anywhere else, so you don't have to keep looking. Things get interesting with collisions but that's a subject for another time.

Re:SeenBefore: A search engine for what you have seen before by genwin on Tuesday September 25, 2012 @01:32AM (#4568781)
OK, thanks, I'm starting to understand. Good explanation. I quit being lazy and searched around too, to see that it's possible.
Re:SeenBefore: A search engine for what you have seen before by StavrosK on Monday September 24, 2012 @09:16PM (#4568001)
Why not use a bloom filter?
Re:SeenBefore: A search engine for what you have seen before by vidarh on Tuesday September 25, 2012 @05:09AM (#4569369)
The main benefit of Bloom filters is that they can be made small. Given that his database takes only 70MB or so and he's not trying to ship this to devices that might have much in terms of space limitations, there would appear to be little point.
Re:SeenBefore: A search engine for what you have seen before by StavrosK on Tuesday September 25, 2012 @05:13AM (#4569380)
Eh, true, I guess redis is sufficiently awesome.
Re:SeenBefore: A search engine for what you have seen before by genwin on Tuesday September 25, 2012 @12:50AM (#4568641)
Maybe because of this fact (according to Wikipedia)?: "The more elements that are added to the set, the larger the probability of false positives."
Re:SeenBefore: A search engine for what you have seen before by StavrosK on Tuesday September 25, 2012 @05:13AM (#4569379)
That depends on its size, though. You can make it larger and get fewer false positives.
Re:SeenBefore: A search engine for what you have seen before by siculars on Monday September 24, 2012 @02:50PM (#4566266)
Wouldn't one simply use one specific browser, say either safari or firefox or chrome, and only that browser for their... unsavory activities? I think that is a great way to keep accounts separate and keep "bad" sites from knowing about "good" sites and vice versa. Just saying. Not that I partake in any such unsavory activities.
Re:SeenBefore: A search engine for what you have seen before by user-id on Monday September 24, 2012 @08:54PM (#4567930)
For testing purposes I use Chrome's "Users" feature to keep an extra profile with no extensions installed handy.

The same could be done for a "Porn" profile too I guess, sand-boxing any history, extensions and bookmarks to that profile. You could even associate tie it to a Google account for portability.

Re:SeenBefore: A search engine for what you have seen before by tammer on Monday September 24, 2012 @07:56PM (#4567742)
This problem is nullified by private browsing. I think the idea is BRILLIANT, as Google's already tracking all my 'legitimate' searches, and I find that most of what I Google are things I've looked at on other machines, or seen already.

The noise introduced by phrasing my query differently is a real problem in search that Google hasn't fixed yet.

Re:SeenBefore: A search engine for what you have seen before by derekorgan on Monday September 24, 2012 @09:42AM (#4564583)
Porn sites are not recorded
Re:SeenBefore: A search engine for what you have seen before by chris24 on Monday September 24, 2012 @11:11AM (#4565035)
How does your system define "porn sites"? What about if it was some porn site no one has ever heard of with an innocent-sounding name/domain?
Re:SeenBefore: A search engine for what you have seen before by vidarh on Tuesday September 25, 2012 @05:14AM (#4569388)
He apprently uses a list of 1.7 million sites. But you can also blacklist sites and have any existing entries for it removed:

https://www.seenbefore.com/blacklist_items