News for hackers. Stuff that matters.

A comment on Git scraping: track changes over time by scraping to a Git repository. Parent comment

Re:Git scraping: track changes over time by scraping to a Gi... by simonw on Thursday August 10, 2023 @07:20PM (#37083002)
Yeah, that's good idea - I need to add that to my suggestions for how to implement this.
[ Parent | on HN ]
Re:Git scraping: track changes over time by scraping to a Gi... by lloydatkinson on Friday August 11, 2023 @05:04AM (#37086601)
It should definitely be optional. I can only imagine some busybody PM insisting they block harmless scrapes.
[ Parent | on HN ]
Re:Git scraping: track changes over time by scraping to a Gi... by pcthrowaway on Friday August 11, 2023 @05:51AM (#37086863)
If you're scraping any significant amount of data (>500K), and depending on the frequency, you might also want to add etag/cache-control headers as well as accept-encoding, to save server bandwidth.

Collecting 1 kB every minute might not be a big deal, but collecting 1 MB every minute would cost an AWS-hosted service >$40/year in additional data transfer costs

[ Parent | on HN ]