News for hackers. Stuff that matters.
Re:Json-Base – Database built as JSON files by oxymoron on Thursday July 02, 2020 @07:06PM (#23719097)
1. A cosmic ray storm turned all ASCII charactees into ECBDIC

2. Lightning struck the 12 V feed and upped the voltage to 10 MV, turning all 0 and 1’s into 6’s

3. Someone spilled a New England Pale Ale on the server

4. The process was assinated by the mysterious killer only known from his modus operandi of leaving OOM written in blood across the syslog

5. Birds nested within the server and fed all the SATA cables to their babies

Seriously though, disk writes aren’t atomic.

Re:Json-Base – Database built as JSON files by lalaland1125 on Thursday July 02, 2020 @08:05PM (#23719526)
File renames are atomic. This is a solved problem:

1. Write your updates to a copy of the file.

2. Do an atomic rename of that copy to the original.

Re:Json-Base – Database built as JSON files by Zarel on Thursday July 02, 2020 @09:53PM (#23720329)
Renames aren't atomic on crash.

https://danluu.com/file-consistency/

Re:Json-Base – Database built as JSON files by mpweiher on Saturday July 04, 2020 @05:55AM (#23731290)
Hmm...unlike the rest of the post, he just asserts this without support.

His sources seem to disagree, certainly with that kind of blanket statement. For example:

Our study takes a pessimistic view of file-system behavior; for example, we even consider the case where renames are not atomic on a system crash.

So this is clearly considered an outlier/unusual.

(e.g., a single 512-byte write or file rename operation are guaranteed to be atomic by many current file systems when running on a hard-disk drive)

[https://www.usenix.org/system/files/conference/osdi14/osdi14...]

I remember reading quite a bit about the (performance reducing) lengths filesystems go to in order to ensure consistency of directory entries even in case of a crash, and for example how "soft updates" were introduced to accomplish the same consistency with less of a performance degradation.

Looking at it from another angle, if you are running on top of a filesystem that cannot keep itself consistent, then you are SOL, there really isn't anything you can do to mitigate.

Just like we can't guarantee that we will be able to persist data that's in memory to disk if the OS is free to kill us at any time. "Best effort" it is, which means getting the data to disk as quickly as possible and not corrupting what is there.

Re:Json-Base – Database built as JSON files by jclulow on Friday July 03, 2020 @12:00AM (#23720895)
They are if you perform the correct sequence of fsync operations on both the file and the directories, and use a file system which is correctly implemented.
Re:Json-Base – Database built as JSON files by chrismorgan on Friday July 03, 2020 @07:13AM (#23722992)
Renames aren’t atomic on crash.
Re:Json-Base – Database built as JSON files by badsectoracula on Saturday July 04, 2020 @04:58AM (#23731084)
What crash?

The article linked above never explains that part, it only assumes that it will happen. From the code it sounds as if the crash can happen in the OS itself (but then the entire kernel will crash). At that point things are completely outside your control and you might as well running on broken hardware.

Re:Json-Base – Database built as JSON files by fanf2 on Thursday July 02, 2020 @11:00PM (#23720658)
1. Write your updates to a copy of the file.

1a. fsync() the file to ensure the contents are durable.

2. Do an atomic rename of that copy to the original.

2a. fsync() the directory to ensure the rename is durable.

Re:Json-Base – Database built as JSON files by fiddlerwoaroof on Friday July 03, 2020 @12:31AM (#23721024)
Or just use a database like SQLite or Postgres that has developers dedicated to solving this problem.
Re:Json-Base – Database built as JSON files by badsectoracula on Saturday July 04, 2020 @04:52AM (#23731068)
Disk writes aren't atomic but you know if they failed or not (unless your OS is lying to you but that is a problem with the OS, not your code).