How to Efficiently Choose the Right Database for Your Applications

Re:How to Efficiently Choose the Right Database for Your App... by bullen on Sunday February 28, 2021 @04:51AM (#26292085)

I would go even further and say:

  +------------+
  | Use a file |
  +------------+

Personally I use JSON over my own async. HTTP (server and client).

[ Parent | on HN ]

Re:How to Efficiently Choose the Right Database for Your App... by jtsiskin on Sunday February 28, 2021 @05:13AM (#26292204)

import json

json = json.dumps(database)

with open("database.json","w") as f:

  f.write(json)
 :)

- 0 dependencies

- easily inspectable and editable with any text editor or cli via jq

- backup and diff

- language agnostic

[ Parent | on HN ]

Re:How to Efficiently Choose the Right Database for Your App... by chousuke on Sunday February 28, 2021 @05:47AM (#26292373)

This "simple" approach will also lead to all kinds of problems quite quickly once you grow past one concurrent user.

If you really must use JSON, at least use SQLite in place of open().

[ Parent | on HN ]

Re:How to Efficiently Choose the Right Database for Your App... by bullen on Sunday February 28, 2021 @06:37AM (#26292639)

A actually use one file per value!

I have to partition the ext4 filesystem with type small othervise I run out if inodes before diskspace!

Here is what it looks like in action: http://root.rupy.se/link/type/task/847068548006606746

The front end: http://talk.binarytask.com

[ Parent | on HN ]

Re:How to Efficiently Choose the Right Database for Your App... by scottlamb on Sunday February 28, 2021 @12:23PM (#26294973)

- occasionally accidentally truncates the database

If you're going to do this, at least write to a temporary file, fsync the file, rename into place, and fsync the directory. But I recommend SQLite any time someone is tempted to write these lines.

[ Parent | on HN ]

Re:How to Efficiently Choose the Right Database for Your App... by uyt on Sunday February 28, 2021 @07:51AM (#26293044)

I tried this approach for scraping a REST API returning json (on a few tens of thousands of ids).

Since it was a long running job with many failures (throttling and connectivity issues), I had to constantly kill and restart the script.

I needed to know what ids to skip over on restart but for some reason listing a folder with a few tens of thousands of files is very slow. So restarting takes forever. I also ran into issues with nonatomic file writes where even though a json file was written, it was incomplete or empty.

I think if I had just inserted them as json strings into sqlite it would've been more robust? I am okay with losing writes (since I will just redownload them), but it was the incomplete writes and long time to reload the set of seen ids on restart that drove me crazy.

[ Parent | on HN ]