Re:Show HN: Till – Unblock and scale your web scrapers, with...
by 1vuio0pswjnm7 on Wednesday August 04, 2021 @06:01PM
(#28066983)
Then Google decided that no one has a right to scrape Google.
The web needs a more efficient system for distributing an index of its content. Having web developers design websites in a gazillion different permutations, when they are all bascially doing the same thing, and then having a handful of companies "scrape" them is neither an efficient nor sensibly-designed system.
The web (more generally, the internet) is a giant copy machine. Google, NSA and others have copies. Yet if we were to allow everyone to have copies by faciltating this through technical means (e.g., Wikipedia-style data dumps), many folks would panic. When Google first started indexing, it was not a business, and many folks did panic and there were many complaints. It's 2021; folks are still spooked by others being able to easily copy what they publish to the web. However it's OK to them if it's Google doing the copying. If there were healthy competition and many search engines to choose from, if one search engine did not have the lion's share of web traffic, it's doubtful Google would be given "exclusive" permission in robots.txt.
The web needs a more efficient system for distributing an index of its content. Having web developers design websites in a gazillion different permutations, when they are all bascially doing the same thing, and then having a handful of companies "scrape" them is neither an efficient nor sensibly-designed system.
The web (more generally, the internet) is a giant copy machine. Google, NSA and others have copies. Yet if we were to allow everyone to have copies by faciltating this through technical means (e.g., Wikipedia-style data dumps), many folks would panic. When Google first started indexing, it was not a business, and many folks did panic and there were many complaints. It's 2021; folks are still spooked by others being able to easily copy what they publish to the web. However it's OK to them if it's Google doing the copying. If there were healthy competition and many search engines to choose from, if one search engine did not have the lion's share of web traffic, it's doubtful Google would be given "exclusive" permission in robots.txt.