Definitely want to see more people doing that kind of low-level work instead of falling back to either 'use elasticsearch' or 'you can't, you're not google'.
For the moment I have just south of 20 million URLs indexed.
1 x 20 million bytes = 20 Mb.
10 x 20 million bytes = 200 Mb.
100 x 20 million bytes = 2 Gb.
1,000 x 20 million bytes = 20 Gb.
10,000 x 20 million bytes = 200 Gb.
100,000 x 20 million bytes = 2 Tb.
1,000,000 x 20 million bytes = 20 Tb.
This is still within what consumer hardware can deal with. It's getting expensive, but you don't need a datacenter to store 20 Tb worth of data.
How many bytes do you need, per document, for an index? Do you need 1 Mb of data to store index information about a page that, in terms of text alone, is perhaps 10 Kb?
How do you rank the results (is it based on content only) or you have external factors too?
What is your personal preferred search option of the 7 and why?
Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper.
>How do you rank the results (is it based on content only) or you have external factors too?
I rank based on a pretty large number of factors, incoming links weighted by the "textiness" of the source domain, and similarity to the query.
> What is your personal preferred search option of the 7 and why?
I honestly use Google for a lot. My search engine isn't meant as a replacement, but a complement.
> Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
Are you kidding? I think the Patreon is a resounding success! I'm still a bit stunned. I've gotten more support and praise, not just in terms of money but also emails and comments here than I could have ever dreamed possible.
And this is just the start, too. I only recently got the search engine working this well. I have no doubt it can get much better. The fact that I have 11 people with me on that journey, even if they "just" pay my power bill, that's amazing.
I'm honestly a bit at a loss for words.
And I am not kidding. I think for something that got so much attention on HN, where realistically this kind of product can only exist for now, the 'conversion' rate was very low. Billion dollar companies were made of HN threads with lot less engagement. Makes me wonder do we really want a search engine like this or we just like the idea of it?
And what are the barriers to use something like this? You say yourself that you are using Google most of the time. Is jumping to check results on this engine going to be too much friction for most uses?
Can something like this exist in isolation? What kind of value would it need to provide for users to remember using it en-masse as an additional/primary vertical search like they do for Amazon?
Just thinking out-loud as I am also interested in the space (through http://teclis.com).
Ultimately I think running something like this for profit would create really unhealthy incentives to make my search engine worse. Any value it brings, right now, it brings because it isn't trying to cater to every taste and every use case.
I also hate the constant "don't forget to slap the like and subscribe buttons"-shout outs of modern social media, even though I'm aware they it is extremely efficient. If I went down that route, I would become part of the problem I'm trying to help cure. I do feel the sirens' call though, it's intoxicating getting this sort of praise and attention.
I want this to be a long-term project, not a some overnight cinderella story.
In the end, my search engine is never going to replace google. It isn't trying to, it's trying to complement it. It's barely able now, but hopefully I can make it much better in the months and years to come.
This allows quite a bit of its own kind of freedom even if maximum financial opportunity is not fully exploited. Perhaps even because you are not grasping for every dollar on the table at all times.
You can do things without having to know if they will pay off, and if it turns out big anyway you can make money as a byproduct of what you do rather than having pure financial pursuit be the root of every goal.
So given your setup is already ideal for 'conversions' for this population (low profile, high integrity, no BS) I was simply genuinely surprised that only 11 people converted given enormous visibility/interest this thread had. Hope that makes sense.
I'd absolutely consider sending someone money if they kept bringing something of value into my life. If I want more people to join the patreon, I'll just have to earn their trust and support.
Plus another excellent feature was you would get the same search results no matter who or where you were for quite some period of calendar time.
If something new did appear it was likely to be one of the new sites that was popping up all the time and it was likely to be as worthwhile as its established associates on the front page.
You shouldn't need to crawl nearly as fast if you can compensate by treading more suitably where those have gone before.
How does the range binary search work, does it just prune out the overlaps, how efficient is it and how much data do you have in there for say "hello" and "world" f.ex?
The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence.
Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8.
The first range corresponds to the URLs 1, 5, 7; and the second 2, 5, 8.
If you search for hello world, it will first pick a range, the range for "hello", let's say (1,5,7); and then do binary searches in the second range -- the range corresponding to "world" -- (2,5,8) to find the overlap.
This seems like it would be very slow, but since you can trivially find the size of the ranges, it's possible to always do them in an order of increasing range-sizes. 10 x log(100000) is a lot smaller than 100000 x log(10)
Funny I also selected "hello" and "world" above! Xo
My system is also written in Java btw!
Here are example results of my word search:
http://root.rupy.se/node/data/word/four
http://root.rupy.se/node/data/word/only
etc.
But yeah, in short pseudocode:
for url in range-for-"hello":
if binary-search (range-for-"world", url):
yield url
I would first search for the bigram hello_world, that's an O(1) array lookup; as then documents merely containing the words hello and world (usually not a good search result), that's the algorithm I'm describing in the parent comment.
A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays.
So when you search for two words, it translates them into IDs using the special hash map, goes to the words file and finds the least common of the words; starts with that.
Then it goes to the words file and looks up the URL range of the first word.
Then it goes to the words file and looks up the URL range of the second word.
Then it goes through the less common word's range and does a binary search for each of those in the range of the more common word.
Then it grabs the first N results, and translates them into URLs (through mariadb); and that's your search result.
I'm skipping over a few steps, but that's the very crudest of outlines.