News for hackers. Stuff that matters.
Re:Git scraping: track changes over time by scraping to a Gi... by alphanumeric0 on Thursday August 10, 2023 @09:40PM (#37084081)
What's the benefit of this versus a time series database?
Re:Git scraping: track changes over time by scraping to a Gi... by simonw on Thursday August 10, 2023 @09:48PM (#37084144)
I'm not sure how those are comparable.

The trick here is to take some source of information online that's updated frequently and turn that into a historic record of every change made to that source, by setting up a GitHub repository and dropping a YAML file into it setting up a scheduled action.

Achieving the same thing with a time series database would require a whole lot more work I think - you'd need to run that database somewhere, then run code that scrapes and writes to it on a scheduled basis.

If you already have a time series database running and a machine that runs cron I guess it wouldn't be too much work to put that in place.

Git scraping also lets you easily track changes made to textual content, which I don't think would fit neatly in a time series database.

Re:Git scraping: track changes over time by scraping to a Gi... by alphanumeric0 on Thursday August 10, 2023 @11:17PM (#37084722)
Yes, maybe they are a bit 'apples to oranges'. You have some good points, especially when it comes to textual data. Thanks!
Re:Git scraping: track changes over time by scraping to a Gi... by fragmede on Friday August 11, 2023 @01:01AM (#37085271)
I mean you could use SQLite the wrong way and use it as a time series database, which would save you from having to have a machine to host it, and I'm sure you could cobble together some sort of hosting for it and glue it to a web cron system. this github seems quite a bit more straightforwards, but then you're on git instead of something else.
Re:Git scraping: track changes over time by scraping to a Gi... by q87b on Friday August 11, 2023 @08:01AM (#37087694)
Tracking changes vs. tracking full versions of data is one immediate benefit (or difference, depending on your thoughts)