I did not do absolutely nothing, I incremented a variable under the lock to hopefully avoid optimizer shenanigans and also tried an unoptimized debug build. Now I tired just spinning and incrementing a variable for a millisecond under the lock, that took 4125 ms for 4 processes with 1000 iterations each. A single process without lock and 4000 iterations took 4005 ms. Those 120 ms would make it a 30 µs locking overhead, the first test without work under the lock hinted at 5 µs. This is probably one of the harder things to profile and depends on quite some factors, but some ten thousand locks per second is probably the correct order of magnitude.

That matches my expectations. Really, if you are doing something as slow as writing logs to a disk, and the number of processes/threads is not in the tens or hundreds, I don't imagine locking overhead is your problem, given the speed of disk storage.

That said, I think the main problem with that is to do it cross platform, which the article goes to paint to mention quite a bit. I imagine the whole point here is to be portable, and I'm not sure what mechanisms work best with that, and what platforms they are available on. I uncovered some unsettling info about Fcntl locking[1], but generally I just used flock when I had to care about it, but i don't think that exists on windows normally(?).

1: http://0pointer.de/blog/projects/locking.html

That was also my first thought, why even use a separate lock and not just the one associated with a file? But I was not sure if that would work, whether it was flexible enough to support a single exclusive writer and multiple readers. I am still not absolutely sure how the option you specify when opening a file and when locking a file later exactly interact, but I am now convinced that it is possible, Windows has LockFileEx [1] and UnlockFileEx [2] which even support limiting the lock to specific blocks within the file.

Another idea was, why not just exclusively open the file, write to it and then closed it again? Why even bother having the file open in several processes at the same time? I am not sure what the overhead would be, but I guess it would not be to terrible. And if you can afford to buffer say 1000 records you want to write, then you can simply cut down the overhead by a factor of 1000 by just doing that.

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

[2] https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

For most purposes LockFileEx and UnlockFileEx can be used in almost the same way as flock and funlock. I recently worked on some code that needed to work on Windows, Linux and Mac and I managed to get the file locking semantics to work equivalently on all platforms.

Regarding having a buffer of records and only locking / unlocking once per batch write, I've used this technique in a program that writes logs to CSV and it works perfectly. You obviously need to tune the buffer size based on the rate and size of new records! One advantage of this is that, depending on the data you're dealing with, you can pre-sort the data in the buffer and end up with mostly sorted (less interleaved) data in the final output. If you need sorted output, then this can dramatically reduce the time taken to sort the final file.