I suspect shared lock performance is the sticking point.

Personally, I agree with tlb: it seems simpler to generate one file per thread, and combine them at the end.

I just tried it on Windows, 4 processes each acquiring the lock 100,000 times and doing no work while holding the lock, and it took about 2000 ms. I don't think that would become the bottleneck for a lot of applications.

EDIT: Corrected the numbers, I had a bug, previously it said 4 processes with 10 million locks per process and 750 ms. It is still pretty fast even if 250 times slower than initially claimed.

I would be interested in seeing what it's like when the locking process sleeps for a small amount of time, and how that affects lock contention and CPU load. Maybe actually doing some work (such as writing the current timestamp 10 times) as well. Unless you actually are doing something, there could be some interesting differences that are covered up or optimized away.

I did not do absolutely nothing, I incremented a variable under the lock to hopefully avoid optimizer shenanigans and also tried an unoptimized debug build. Now I tired just spinning and incrementing a variable for a millisecond under the lock, that took 4125 ms for 4 processes with 1000 iterations each. A single process without lock and 4000 iterations took 4005 ms. Those 120 ms would make it a 30 µs locking overhead, the first test without work under the lock hinted at 5 µs. This is probably one of the harder things to profile and depends on quite some factors, but some ten thousand locks per second is probably the correct order of magnitude.

That matches my expectations. Really, if you are doing something as slow as writing logs to a disk, and the number of processes/threads is not in the tens or hundreds, I don't imagine locking overhead is your problem, given the speed of disk storage.

That said, I think the main problem with that is to do it cross platform, which the article goes to paint to mention quite a bit. I imagine the whole point here is to be portable, and I'm not sure what mechanisms work best with that, and what platforms they are available on. I uncovered some unsettling info about Fcntl locking[1], but generally I just used flock when I had to care about it, but i don't think that exists on windows normally(?).

1: http://0pointer.de/blog/projects/locking.html

That was also my first thought, why even use a separate lock and not just the one associated with a file? But I was not sure if that would work, whether it was flexible enough to support a single exclusive writer and multiple readers. I am still not absolutely sure how the option you specify when opening a file and when locking a file later exactly interact, but I am now convinced that it is possible, Windows has LockFileEx [1] and UnlockFileEx [2] which even support limiting the lock to specific blocks within the file.

Another idea was, why not just exclusively open the file, write to it and then closed it again? Why even bother having the file open in several processes at the same time? I am not sure what the overhead would be, but I guess it would not be to terrible. And if you can afford to buffer say 1000 records you want to write, then you can simply cut down the overhead by a factor of 1000 by just doing that.

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

[2] https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

For most purposes LockFileEx and UnlockFileEx can be used in almost the same way as flock and funlock. I recently worked on some code that needed to work on Windows, Linux and Mac and I managed to get the file locking semantics to work equivalently on all platforms.

Regarding having a buffer of records and only locking / unlocking once per batch write, I've used this technique in a program that writes logs to CSV and it works perfectly. You obviously need to tune the buffer size based on the rate and size of new records! One advantage of this is that, depending on the data you're dealing with, you can pre-sort the data in the buffer and end up with mostly sorted (less interleaved) data in the final output. If you need sorted output, then this can dramatically reduce the time taken to sort the final file.

The issue is the contention caused by all the processes waiting on file IO behind the lock, not the locking

You don't actually need to hold the lock while doing I/O - just while allocating yourself a region of the file that's yours to exclusively write to (eg. by opening without O_APPEND and instead using pwrite(2) to write at a specific offset).

If I/O were the bottleneck then that would be the case with or without a lock, wouldn't it?

It depends on what has to wait for the IO to complete. ETW, for example, is a Windows-wide event framework and it would be untenable for everything using it to wait on the file IO of everything else making use of it. So, it has a solution that makes use of locks but nothing ever has to wait on file IO.

File IO is always expensive, the trick is always in how you work around that.

That does seem simpler. I've run into variants of this problem where each thread/process can potentially produce GBs of data. For that use case performance would suffer if combining were done as a separate step.

Actually, since the data is sequential, I think you can write a very quick and optimized combining process.

E.g.

Open each file and buffer a small amount, such as 4k.

1) Read the first record of each buffer

2) Display the first record of the buffers

3) Read the next record from the buffer you just consumed from, and re-buffer another 4k from it if needed.

4) Go to 2.

Go ahead and change how much you want to buffer from each file depending on what seems to work well and to take advantage of sequential read times if using media that uses that to it's advantage (spinning disks), but I don't see that being very slow. Worst case, you are naively sorting X records each time to find the first, but since you are always consuming them in order, you can easily keep track of the current order and do a dingle comparison per loop after the first.

Namely，MapReduce

Or even older - an external merge sort (https://en.wikipedia.org/wiki/Merge_sort#Use_with_tape_drive...). It was designed for doing out of memory sorts backed by a much slower medium (like disk).

There may be no locking mechanism available. One example is multiple computers writing to a file on an NFS share with the nolock mount option.

> multiple computers writing to a file on an NFS share with the nolock mount option.

Yes, but that's the kind of thing everyone explicitly tells you never to do.