News for hackers. Stuff that matters.
Re:Share code that uses new URL Search tool and win AWS credit by djoerd on Tuesday March 05, 2013 @03:41PM (#5327193)
While I know that some of the pages of my home page are in the crawl, they do not show up with the following query: http://urlsearch.commoncrawl.org/?q=nl.utwente.cs.wwwhome%2F... nor with: http://urlsearch.commoncrawl.org/?q=nl.utwente.cs.wwwhome%2F... (no, this is not only an ego search problem ;-) )
Re:Share code that uses new URL Search tool and win AWS credit by anjackson on Tuesday March 05, 2013 @03:47PM (#5327244)
Yes, that's a little odd. If you shorten the search term, the results come up just fine: http://urlsearch.commoncrawl.org/?q=nl.utwente.cs.wwwhome/~h...
Re:Share code that uses new URL Search tool and win AWS credit by greglindahl on Tuesday March 05, 2013 @05:05PM (#5327693)
Works on me!

http://urlsearch.commoncrawl.org/?q=com.pbm.www%2F~lindahl

So there's a bug there, but not all the time for ~.

Re:Share code that uses new URL Search tool and win AWS credit by LisaG on Tuesday March 05, 2013 @05:30PM (#5327850)
I hope that some of you who use/play around with the Common Crawl data will try out using the JSON files from the URL Search and then share your code.

If you didn't see the details in the blog post, Common Crawl is giving out $100 in AWS credit to the first five people who share code that incorporates a JSON file from the URL Search.

Re:Share code that uses new URL Search tool and win AWS credit by visarga on Wednesday March 06, 2013 @01:42AM (#5329739)
Is it possible to get a list of webhosts, like all the domains and subdomains, stripped from the rest of the url?
Re:Share code that uses new URL Search tool and win AWS credit by LisaG on Tuesday March 05, 2013 @03:32PM (#5327137)
From @djoerd Why does @CommonCrawl URL search (http://urlsearch.commoncrawl.org/ ) need 'tld.domain' format rather than 'domain.tld'? Read Google's BigTable paper.
Re:Share code that uses new URL Search tool and win AWS credit by frederi on Tuesday March 05, 2013 @03:35PM (#5327163)
Why can't they just write code that reverses the input?
Re:Share code that uses new URL Search tool and win AWS credit by srobertson on Tuesday March 05, 2013 @04:04PM (#5327327)
The main intent of the search is to retrieve a list of urls that the site crawled for a given subdomain, domain or tld. So for now you can do that using reversed url notation. Which I admit is not very intuitive.

We're toying with the idea of implementing some sort of wild card that way we can present the urls in natural order. Something like *.google.com to retrieve all urls under google. But we wanted to judge the level of interest first. After all "done" is better than "perfect".

Re:Share code that uses new URL Search tool and win AWS credit by LisaG on Tuesday March 05, 2013 @04:10PM (#5327360)
"Done" is better than "perfect" should be on a sign hanging in every startup.
Re:Share code that uses new URL Search tool and win AWS credit by Aloisius on Tuesday March 05, 2013 @03:46PM (#5327234)
Just an oversight. Most of our work is done by people who graciously volunteer their time. We'll get that fixed.
Re:Share code that uses new URL Search tool and win AWS credit by lubujackson on Tuesday March 05, 2013 @03:47PM (#5327248)
I'd love it if there was a feature to search for a specific URL. Like if "com.google" just loaded the Google homepage if you put it in quotes.
Re:Share code that uses new URL Search tool and win AWS credit by srobertson on Tuesday March 05, 2013 @03:58PM (#5327300)
good suggestion
Re:Share code that uses new URL Search tool and win AWS credit by lubujackson on Tuesday March 05, 2013 @03:52PM (#5327270)
Top results for "com" are a little odd. Seems like @ wasn't filtered from the domain part of the URL (though it should be, I would think).
Re:Share code that uses new URL Search tool and win AWS credit by srobertson on Tuesday March 05, 2013 @03:57PM (#5327298)
I think it's just that the site converts unicode urls for display purposes. If you click on one of the links with "@" in it, you'll see the real url in url encoded format

http://%2E%2E%[email protected]/Ajax/get.comments/oid/4612 176.34.181.212 20120516214328 text/html 912

Re:Share code that uses new URL Search tool and win AWS credit by djoerd on Tuesday March 05, 2013 @03:42PM (#5327204)
The first FAQ link seems to be broken (maybe a web server setting gone bad?) BTW, this is a great resource. Thanks for sharing this!
Re:Share code that uses new URL Search tool and win AWS credit by LisaG on Tuesday March 05, 2013 @03:45PM (#5327227)
Thanks for the catch Djoerd! We will fix it now