pablohoffman

Hey folks, I'm the one interviewed, happy to answer any follow up question here.

Not surprised, I'm pretty low profile :)

Portia from Scrapinghub is 100% open source: http://scrapinghub.com/portia/

We released a similar open source tool for visual scraping, earlier this year, called Portia: https://github.com/scrapinghub/portia

It's been getting quite a bit of traction and we're currently working on the integration with Scrapinghub platform (disclaimer: I work there) for those who prefer a hosted version.

Not yet.

Yes, you just need to select a different field type ("text", instead of "html").

We are using CDH4.2 and have had a very positive experience so far.

Cloudera has in fact been an inspiration for us to follow, you guys have really struck the right balance between open source and commercial support. We follow the same philosophy with Scrapy (an open source web crawling framework), as you do with Hadoop and its ecosystem.

We are not plainly complaining about MongoDB, nor saying it's useless. We are just explaining why it's a poor choice for a specific use case: storing scraped data.

FWIW, we still use Mongo in other internal applications, it's just not the right choice for our crawl data storage backend.

We went with HBase. Cassandra would have been suitable too, but we already use Hadoop for data processing so it was a natural choice within the infrastructure ecosystem. We will write a followup about that.

The title of this post is misleading, it makes you think GitHub is actually broken, when the article is about complaining how their fork feature works.

I initially submitted this post, but then deleted it and resubmitted to the original post on Common Crawl blog: http://news.ycombinator.com/item?id=3208853

I now regret since this one got much more attention. I was under the impression that linking to the original post was more welcomed here HN, but it seems this is not always the case.

I found strange that no one mentioned Privnote (https://privnote.com), a much older/popular service that does this but more securely since it generates URLs with keys in the fragment to encrypt the messages on client side (so that the plain message never hits the server).

Did it work?

I've been using tiled window managers for 7 years now (currently on awesome, which I switched 3 years ago) and the interface has always remained pretty consistent. I wonder if Linus has tried tiled WMs?

After checking a few registrars, I found they only sell third-level domains (.com.ng, .net.ng), not second-level domains (like "amazi.ng") which is probably what most people here would be interested in.

Example: http://www.webdomains.com.ng/myaccount/whois.php?step=1

It's a shame they're missing the opportunity to sell such a nice top-level domain - or perhaps it's done on purpose?

Someone already wrote a Delicious scraper in Scrapy (Python): http://news.ycombinator.com/item?id=2015680

Feeling sad would imply that you hold some respect to the judges that decided who won, but I think Time magazine just lost the respect of many readers. I'm kinda sad for them, not for Assange.

That reminded me of a recent Paul Graham tweet:

  You've found market price when buyers complain but still pay.

http://twitter.com/#!/paulg/status/22576762202

Primary source thread: http://news.ycombinator.com/item?id=1849274

Is it me, or Zed is starting to get annoyed at the Python community now?. I wonder when he'll write "Python is a Ghetto".