News for hackers. Stuff that matters.
Re:IndexTank / 80legs Crawlathon (developer contest) by va_coder on Wednesday June 15, 2011 @03:23PM (#2658632)
IndexTank as opposed to WebSolr has a proprietary api. I don't think you can easily move the data to your own servers like you can with WebSolr
Re:IndexTank / 80legs Crawlathon (developer contest) by diego on Wednesday June 15, 2011 @05:29PM (#2659102)
Yes you can! Our indexes are persisted in Lucene format internally. Nobody has asked to download an index but if they asked we'd make it available.
Re:IndexTank / 80legs Crawlathon (developer contest) by personalcompute on Wednesday June 15, 2011 @03:35PM (#2658685)
Another problem is 80legs will only crawl pages <100kb with the free/contest plan. This means it is useless for all but the smallest pages.
Re:IndexTank / 80legs Crawlathon (developer contest) by jdrock on Wednesday June 15, 2011 @03:42PM (#2658712)
Er.. not true, the contest plan (which is different from the free plan) allows up to 10 MB download. Registered contestants should have access to their plans within an hour of registering.

And if you don't have it, just contact us: http://www.80legs.com/contact.html.

Re:IndexTank / 80legs Crawlathon (developer contest) by personalcompute on Wednesday June 15, 2011 @08:54PM (#2659699)
Ah, awesome, thank you for the correction, this allows me to continue with my plan. I said that within an hour of registering, but had asked on their support page as well.
Re:IndexTank / 80legs Crawlathon (developer contest) by revorad on Wednesday June 15, 2011 @04:49PM (#2658965)
If I'm building a search engine, say focused on telescopes, can I use 80legs to crawl youtube videos (or results from google video search)? I mean can I add this url to my list of urls to crawl? - http://www.youtube.com/results?search_query=telescope
Re:IndexTank / 80legs Crawlathon (developer contest) by jdrock on Wednesday June 15, 2011 @05:03PM (#2659014)
Our default crawler obeys robots.txt and it looks like the /results URLs are not allowed. However.. I think you could start from a URL like http://www.youtube.com/watch?v=sAzhOSbxMiI and then crawl to the linked videos from there...
Re:IndexTank / 80legs Crawlathon (developer contest) by revorad on Wednesday June 15, 2011 @05:16PM (#2659053)
Oh I didn't know about the robots.txt rule for /results. Thanks for pointing it out or I would have built my own crawler and got banned. I think I'll go with playlists.
Re:IndexTank / 80legs Crawlathon (developer contest) by orborde on Wednesday June 15, 2011 @01:20PM (#2658055)
What's the goal of the contest? "Build the most awesome app on our platform"?

It's not clear to me reading the post.

Re:IndexTank / 80legs Crawlathon (developer contest) by diego on Wednesday June 15, 2011 @01:45PM (#2658203)
Create a small web search engine. Crawl stuff with 80 legs, index it with IndexTank. Example:

- Crawl everything you can find about bitcoin and do a bitcoin search engine (just popped up in my mind, may not be the most interesting idea).

Re:IndexTank / 80legs Crawlathon (developer contest) by personalcompute on Wednesday June 15, 2011 @02:16PM (#2658351)
Then post the bitcoin-related whatever to HN. Instant winner.
Re:IndexTank / 80legs Crawlathon (developer contest) by btucker on Wednesday June 15, 2011 @03:05PM (#2658561)
Is there any documentation on how to get started using these two products together?
Re:IndexTank / 80legs Crawlathon (developer contest) by diego on Wednesday June 15, 2011 @03:20PM (#2658619)
It should be pretty straightforward:

- run a crawl at 80legs - download the results as a csv or xml - feed them to IndexTank using your client of choice

Over the weekend I'll put on my (rusty) hacker hat, do an example and blog about it.

Re:IndexTank / 80legs Crawlathon (developer contest) by jdrock on Wednesday June 15, 2011 @03:24PM (#2658641)
Specific documentation for 80legs is available at http://wiki.80legs.com. To get fancy, you may want to check out http://wiki.80legs.com/80apps to make your own custom extractors. Note that any custom extractors you write will output files in binary in 80legs, so you'll need to convert the byte array to .txt, .csv or .xml or whatever format you want.

(We post custom extractor results as binary because you can also return stuff like images!)

Re:IndexTank / 80legs Crawlathon (developer contest) by Karhan on Thursday June 16, 2011 @01:35PM (#2662318)
Is there a good reason that reddit has banned 80legs crawling?
Re:IndexTank / 80legs Crawlathon (developer contest) by jdrock on Thursday June 16, 2011 @02:31PM (#2662558)
True story: We crawled them a while back (before they expanded their engineering team) and because of our distributed system, "alarms" were going off. Rather then take time to tell their system we were not a DDOS attack, they put us in robots.txt. I imagine their small team had other stuff to work on.
Re:IndexTank / 80legs Crawlathon (developer contest) by xtacy on Friday June 17, 2011 @10:20AM (#2665574)
So, 80legs caused more traffic volume to reddit than Google crawler? That seems interesting; why would it be the case?
Re:IndexTank / 80legs Crawlathon (developer contest) by jdrock on Friday June 17, 2011 @11:20AM (#2665825)
Not necessarily more traffic, though that's possible. More likely it was the fact that it was coming from multiple IP addresses.
Re:IndexTank / 80legs Crawlathon (developer contest) by Omnipresent on Wednesday June 15, 2011 @01:29PM (#2658105)
can't sign up yet. Seems to be down

80legs is currently down for maintanence.

We're working on upgrades, new releases, and other fun stuff! Be back soon!

Re:IndexTank / 80legs Crawlathon (developer contest) by jdrock on Wednesday June 15, 2011 @02:14PM (#2658345)
Hm.. can you try again? Seems to be working from our end!