Re:How to Efficiently Choose the Right Database for Your App...
by uyt on Sunday February 28, 2021 @07:51AM
(#26293044)
I tried this approach for scraping a REST API returning json (on a few tens of thousands of ids).
Since it was a long running job with many failures (throttling and connectivity issues), I had to constantly kill and restart the script.
I needed to know what ids to skip over on restart but for some reason listing a folder with a few tens of thousands of files is very slow. So restarting takes forever. I also ran into issues with nonatomic file writes where even though a json file was written, it was incomplete or empty.
I think if I had just inserted them as json strings into sqlite it would've been more robust? I am okay with losing writes (since I will just redownload them), but it was the incomplete writes and long time to reload the set of seen ids on restart that drove me crazy.
Since it was a long running job with many failures (throttling and connectivity issues), I had to constantly kill and restart the script.
I needed to know what ids to skip over on restart but for some reason listing a folder with a few tens of thousands of files is very slow. So restarting takes forever. I also ran into issues with nonatomic file writes where even though a json file was written, it was incomplete or empty.
I think if I had just inserted them as json strings into sqlite it would've been more robust? I am okay with losing writes (since I will just redownload them), but it was the incomplete writes and long time to reload the set of seen ids on restart that drove me crazy.