Yeah I have a bunch of these using pretty-printed JSON - here's one that scrapes Hacker News for mentions of my site, for example: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...
Re:London Street Trees
by keepamovin on Friday September 08, 2023 @01:47AM
(#37429854)
Wow you're the creator of datasette! Cool, man! I thought that was a really revolutionary idea, and also related to this notion of git backed DB...I'm so glad to sort of see it confirmed with this! Haha :)
I've been thinking about this recently also. I can't think of any downsides off the top of my head but I might be blinded by the idea? Can you?
I've had dozens of repos doing this kind of thing for a few years now and I've not found any downsides yet - it just works.
There's a size limit to consider: GitHub won't let you push a file larger than 100MB and prefer you to keep the repo itself below 2GB.
Most of my scraping projects are a tiny fraction of that size though.
I explored more of this "git as DB backend" in some places including: https://github.com/dosyago/sirdb
Also, just as a headsup to any folks, the SF version of this "tree map" (Heh) is at: https://web.archive.org/web/20230328192805/https://bsm.sfdpw... (O site and archive seems to be down)