Re:Git scraping: track changes over time by scraping to a Gi...
by simonw on Friday August 11, 2023 @01:43AM
(#37085478)
Oh, I see what you mean.
Yeah, there's definitely a lot to be said for breaking up your scraped data into separate files rather than having it all in a single file.
I have a few projects where I do that kind of thing. My best example is probably this one, where I scrape the "--help" output for the AWS CLI commands and write that into a separate file for each command:
help-scraper/tree/main/aws: https://github.com/simonw/help-scraper/tree/main/aws
This is fantastically useful for keeping track of which AWS features were added or changed at what point.
Rather than a JSON with a bunch of weather stations, make a directory with a bunch of stations as subdirectories, each with lat, long, temp, humidity properties. Let the fs express the structure.
Then when we watch in git, we can filter by changes to one of these subdirs, for example. Or see every time the humidity changes in one. I don't have a good name for the general practice, but trying to use the filesystem to express the structure is the essence.