Re:Git scraping: track changes over time by scraping to a Gi...
by simonw on Friday August 11, 2023 @01:43AM
(#37085478)
Oh, I see what you mean.
Yeah, there's definitely a lot to be said for breaking up your scraped data into separate files rather than having it all in a single file.
I have a few projects where I do that kind of thing. My best example is probably this one, where I scrape the "--help" output for the AWS CLI commands and write that into a separate file for each command:
Yeah, there's definitely a lot to be said for breaking up your scraped data into separate files rather than having it all in a single file.
I have a few projects where I do that kind of thing. My best example is probably this one, where I scrape the "--help" output for the AWS CLI commands and write that into a separate file for each command:
help-scraper/tree/main/aws: https://github.com/simonw/help-scraper/tree/main/aws
This is fantastically useful for keeping track of which AWS features were added or changed at what point.