News for hackers. Stuff that matters.

A comment on Git scraping: track changes over time by scraping to a Git repository. Parent comment

Re:Git scraping: track changes over time by scraping to a Gi... by RockRobotRock on Thursday August 10, 2023 @08:18PM (#37083492)
It's tough when there's a cat and mouse game to spoof your UA so you don't get blocked. I wish webmasters had better relationships with scrapers and could accept the realities that your data will be scraped no matter how much you try and stop it.
[ Parent | on HN ]
Re:Git scraping: track changes over time by scraping to a Gi... by hosteur on Friday August 11, 2023 @07:44AM (#37087572)
IMO, We should really just get rid of the user agent header altogether.
[ Parent | on HN ]