If you really must use JSON, at least use SQLite in place of open().
I have to partition the ext4 filesystem with type small othervise I run out if inodes before diskspace!
Here is what it looks like in action: http://root.rupy.se/link/type/task/847068548006606746
The front end: http://talk.binarytask.com
If you're going to do this, at least write to a temporary file, fsync the file, rename into place, and fsync the directory. But I recommend SQLite any time someone is tempted to write these lines.
Since it was a long running job with many failures (throttling and connectivity issues), I had to constantly kill and restart the script.
I needed to know what ids to skip over on restart but for some reason listing a folder with a few tens of thousands of files is very slow. So restarting takes forever. I also ran into issues with nonatomic file writes where even though a json file was written, it was incomplete or empty.
I think if I had just inserted them as json strings into sqlite it would've been more robust? I am okay with losing writes (since I will just redownload them), but it was the incomplete writes and long time to reload the set of seen ids on restart that drove me crazy.
json = json.dumps(database)
with open("database.json","w") as f:
- 0 dependencies- easily inspectable and editable with any text editor or cli via jq
- backup and diff
- language agnostic