The Archive.org crawl is way too old to be of any use. And the Common Crawl corpus is only marginally bigger than what I crawl myself. There are currently just over a billion pages in my search-index and I can recrawl those about every 45 days. Crawling right now at about 650 URLs/s and 200mbit/s. Only about 60% of URLs crawled end up in the index. The rest are errors, timeouts, redirections, etc.
I have been thinking about adding an API. Rate-limiting is a must as you have correctly said. I'm already in a constant fight against SEOs who try to scrape the search-results and who apparently don't see anything wrong in making 10+ queries/second.
The API would definitly not be a free-for-all, but sort of like "send me an email and describe what you want to do with it, and if I like it I'll give you an API-key".Together with a primer on what not to do of course. :) If people do more than a certain amount of queries/month, then it would also have to be paid.
I figured old data was uninteresting, but TIL that crawling data yourself is not impossible. I mean you have to come from a thousand IPs, browser UAs and cookie states to find everything thanks to the sad state of things, but wow, I didn't realize 200Mbps could do so much. :D
As for the errors et al, that makes me think, I wonder what would happen if you saved that data and folded it into the seed list (with maybe 3 or 4 URLs up the chain leading to that link) for next run.
Agh... so you already have leeches :( all I can suggest is maybe mixing up your HTML, or maybe putting your site behind CloudFlare and, if there's an option(?), setting the attack detection sensitivity all the way up. I have no idea if they can do this.
10 queries/sec... that's... not going to be too great on the hardware :( I mean these people can just crawl the info themselves and get more up-to-date content >.< wow
By "free" API, I don't mean free-for-all "here, have at my bandwidth/hardware" xD - I mean more in the sense that curious shy CompSci types can maybe autogenerate themselves a basic-level API key to experiment with ultra-low-rate data requests here and there, and request a threshold upgrade on their key if they think they have an interesting/justifiable use for the data.
And it would be great if this could be a revenue stream! Have you ever featured the site on HN?
PS. Constructive criticism on design: I would personally alter the page styles a little; it currently looks nothing like what I associate "search engine homepage" with based on what I've learned to expect, and the page style, while very nice, makes me feel like the site is a blog, not a dynamic search engine.
I use the Alexa top-1-million sites as seed-list for the crawler. The errors that do appear during the crawl are either sites that have an outage or more likely simply dead-links. Oh, and URLs that turn out to be blocked by robots.txt. There are a lot of sites out there which block anything but Google and Bing from crawling them.
Cloudflare is not an option for me. It would let Cloudflare know what my users are searching for. VERY big no-no. :)
I can filter out 99% of automated queries. Luckily they are still pretty dumb at the moment and give me enough fixed clues to identify them.
I like your idea of keeping the API free with a very low request-rate. That could work. I would have to find a way that they can't just generate many API-keys though. Using captchas for API-key requests won't stop them from doing that.
I posted a "Show HN" about a year ago. Brought in about 1500 extra visitor that day. Got up to 9th place on the HN homepage that day.
New webdesign is already done. I have a German site too. https://deusu.de which actually gets 90% of traffic. That site already has the new design.
I didn't think of CloudFlare being able to see the traffic... and wow, I never even processed that aspect of their service. But of course...
How good is Google's "[ ] I'm not a robot" checkbox thingy at weeding out bots? And perhaps you could use multiple captcha systems...? (Or are actual people tasked to do signups?!)
I shudder to think of such an idea, but linking API keys to <popular login-with/connect-with-this-site API> may be an alternative. (One thing that comes to mind is that, if someone authenticates using Reddit - which they can do without releasing any account info - is that you could check their (public, but unfakeable) karma counts and use that as a measure of confidence, in addition to the standard account age metric used everywhere.)
The new design is nice :D
And if it's been a year (!), another Show HN sometime would certainly be fine.
I really like obscure search engines, because their (comparatively :P) tiny indexes mean they'll return significantly different data to what say Google or Bing would return.
If there's one thing I would definitely vote for with projects like this, it would be... an API. In a world where virtually every search engine is built around the idea of controlled, non-open access, a predictable, documented, free-access API would be a breath of fresh air, and you'd get people seriously interested in this even though it's small. (You'd definitely need rate limiting though because there are always the hangers-on with the low IQs. >.>)
You may (or may not, no idea) also be interested in the Common Crawl corpus - http://commoncrawl.org/the-data/get-started/ - which contains a few PB of data collected over the past few years (sorted by date). Access via HTTP is only moderately involved: read the line immediately above "Data Format."
Of potentially much less interest is a 2012 80TB snapshot Archive.org are (or at least were) offering; they seeded their indexer off the Alexa top 1m and boxed up whatever came back. https://blog.archive.org/2012/10/26/80-terabytes-of-archived... It's apparently only available on request (I expect a search engine would definitely qualify, especially considering the data is 4 years old now - although I definitely want to see what's in it if you do decide to chase this - serious :P).
Finally, search for "cheetah project" - the 2nd top-level result is messed up, apparently the Chinese (Unicode) got scrambled. Just a note; you may already be aware.