Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots
by rmbyrro on Thursday October 24, 2024 @08:57AM
(#41935051)
Sure, that's why I said "superficially looking", because the back of my head said you obviously have a conscious intention consistent overall.
Would you mind elaborating more on where you draw the line between "good" and "bad" behavior when it comes to scraping?
Is it obeying robots.txt, the monetary purposes (profit / non-profit)?..
I personally have a hard time accepting that a website can allow Google to scrape all their content, but prevent others. In my view, this is anti-competitive behavior favoring a large and evil corporation. Argumenting that Google delivers value back in the form of traffic is unacceptable to me, because it goes against basic principles of net neutrality, which I support.
I'm intetested in reading your thoughts on this.
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots
by cookiengineer on Thursday October 24, 2024 @12:08PM
(#41936891)
I wanted to reply here, and actually posted the comment but then deleted it because some 4chan folks were already starting to comment here again.
I wrote you an email, would love to discuss this further.
Being able to use the web anonymously is a value that should be held up against moral values. Malicious scraping of websites should be, too. Both ideas can simultaneously true, they don't have to be mutually exclusive. I also support initiatives like the web archive which I consider "good behavior" of web scraping.
If a person asks me for the dataset of my website and they don't have a competitive business that they run, I'm even happy to open source the dataset generation part. Contributors get more rights, abusers get less rights. That's how it should be in my opinion.
I don't see them as contradictory.