News for hackers. Stuff that matters.
Re:Common Crawl announces Open Source Big Data code contest ... by bollacker on Tuesday September 18, 2012 @05:42PM (#4540679)
I like how this contest shows that anyone can ask questions that previously only Google, Microsoft (Bing) and a handful of 2nd tier search engines could ask. Now if we can just get a simple query language for it all. I could then pump in $100 in quarters, ask my question, and wait a couple hours. Isn't this what the information ask was supposed to get us?
Re:Common Crawl announces Open Source Big Data code contest ... by 5c4r3d on Tuesday September 18, 2012 @02:28PM (#4539698)
Linking Entities to Wikipedia is awesome. I love the idea of Online Sentiment Towards Congressional Bills but it's too bad they didn't show their results.
Re:Common Crawl announces Open Source Big Data code contest ... by awavering on Tuesday September 18, 2012 @02:34PM (#4539743)
Don't worry, we're working on it - we should have our results up by this weekend. We're planning on doing something like http://www.albertwavering.com/projects/commoncrawl/bill.html to show our results, but I would love to hear suggestions.
Re:Common Crawl announces Open Source Big Data code contest ... by 5c4r3d on Tuesday September 18, 2012 @06:09PM (#4540794)
Oh cool. How many bills are you going to show? Will there be histograms or some kind of visualization or just the lists?
Re:Common Crawl announces Open Source Big Data code contest ... by awavering on Wednesday September 19, 2012 @10:03AM (#4543554)
We are looking at about 50 bills this time around. We really wanted to do a histogram, but we didn't have time to solve the problem of distinguishing between when things are crawled and when things are published.
Re:Common Crawl announces Open Source Big Data code contest ... by LisaG on Tuesday September 18, 2012 @02:42PM (#4539794)
So cool you are putting together a presentation of results!!
Re:Common Crawl announces Open Source Big Data code contest ... by chrishan on Tuesday September 18, 2012 @09:57PM (#4541633)
Glad you like my linking entities to wikipedia entry. Still need some extra work to make a complete/ready-to-use corpus out of the 5 billion webpages
Re:Common Crawl announces Open Source Big Data code contest ... by rjurney on Tuesday September 18, 2012 @05:55PM (#4540738)
The YC crowd needs to pay attention to this. There will be a new wave of startups based on the common crawl as it develops.
Re:Common Crawl announces Open Source Big Data code contest ... by frederi on Tuesday September 18, 2012 @02:35PM (#4539751)
Is the "Facebook infection" code open source? There is no link to the code.