This is basically how Event Tracing for Windows (ETW) solves the problem very efficiently. It uses InterlockedIncrement() to quickly and with minimal contention have something to merge all the records at the end.

If you're on Windows you should use ETW instead of reinventing it when possible :)

ETW is awesome. Too bad it's been plagued by such an opaque API and all the registration junk. I hear it can be easier to use now, but I'm still not sure if an exe can emit ETW events without having to register every possible event or whatever.

The new ETW format is self-documenting at the expense of some overhead. You can set up hello world in ETW in about three lines of code [1], just create the logging channel and log a string or property bag to it.

1. https://blogs.windows.com/buildingapps/2016/06/10/using-devi...

I separate merge step to produce a single file would tripple the transfered data volume which might or might not be an issue. But in case the ordering of the records does not matter as suggested in the article and if you can get the consumer to consume several files, it would indeed be the easiest solution to just have several files and then consume then one after another.