Google in fact does honor requests not to scrape websites, and honors them at Internet scale. Padmapper could, meanwhile, reasonably be defined by their unwillingness to stop scraping a single site whose lawyers made it clear they did not want to be scraped. So I'm not sure your comparison is valid.

Maybe we'd still be using something opt-in like Yahoo Directory, and maybe the web would be better for it. Yahoo directory was slower but in many ways higher-quality; it would've kept the web more decentralized (e.g. Wikipedia would not be the top result for half of queries). Of course there would have been short-term costs to that route.

"decentralized" doesn't magically mean better! I personally believe wikipedia is one of humanity's greatest accomplishments. It's been an enormous group effort.

I have a copy in my phone, unbelievable amounts of knowledge in my pocket, several libraries full. Would it somehow be better if information were harder to find?

I think downranking Wikipedia would make some information easier to find. Often a subject-matter-specific site has a better page than Wikipedia on a given topic - but Wikipedia has more google-juice, so people are directed there instead.

There is certainly value in presenting everything via a consistent interface, as Wikipedia does. But there's also value is subject-specific interfaces. And I find Wikipedia's notability criteria very problematic; Wikipedia can't be treated as a sole source of truth when it simply refuses to include any information about certain subjects. (E.g. I wanted to know about a band I'd seen once, so I looked them up on Wikipedia only to find no page; I wondered if I'd spelled them wrong or some such. Then I looked them up on TV Tropes and found a useful page about them.)

Since everyone here liked the Padmapper UI over Craigslist they were OK with what they were doing, but if this was happening to an HN friendly business the story would be entirely different and there would a massive outcry. The hypocrisy is at least to me highly entertaining.

Very! Google provides a way for you to opt out. This is not the case of "Oh, I went to their website and they sued us out of existence." This is more of a case of they were asked to not do something and they said fuck you I'll do it anyway. I don't like CFAA but I'd say these 3tap people failed at being nice to others.

Screw them.

Yes, I remember those discussions here in HN. The simple fact was they had built and entire product around someone else's data. They maintained an "f-you" attitude even after the cease and desists. Those companies got hammered because of their arrogance. They could have built a CL-killer but instead they built a CL-piggybacker but did so in a way that violated CL's terms of use.

By "somebody else's data" you mean the end users who list their properties on Craigslist, right?

People on HN often complain about YC backed startups if they do morally questionable stuff (see for example any thread about that installer bundling startup whose name I can't recall right now). The HN community isn't as bad as you make it seem.

HN is critical of YC startups.

The only thing you can't do here is criticize Elon Musk.

Scribd and Quora get called on their bullshit all the time.

Dont forget the hubub over YC investing in that adware bundler company (InstallMonetizer), couldnt stop hearing about that for weeks[0]. I certainly think there is a trend towards being positive towards YC properties on HN (surprise!) but if they do something bad they will be excoriated just like the rest.

[0]https://www.techdirt.com/articles/20130115/17343321692/why-a...

Reddit too, given yesterday's news.

Yes. And there are tons of other startups and probably also YC companies involved in heavy scraping, aggregating and selling that information to other folks.