Re:Appending to a file from multiple processes
by DougWebb on Wednesday August 03, 2016 @10:19PM
(#12222952)
It's fairly simple to write a little server that accepts socket connections from multiple processes, and reads a line of data from each connection whenever there is data available and append it to the log file. I wrote a server like that years ago; a write-only socket service which hundreds of separate processes could write high-volume statistics-tracking to. The heart of it was only a few dozen lines.
[ Parent
| on HN ]
Re:Appending to a file from multiple processes
by ww520 on Thursday August 04, 2016 @12:38AM
(#12223417)
Something like syslogd?
Re:Appending to a file from multiple processes
by DougWebb on Thursday August 04, 2016 @09:25AM
(#12225299)
Similar, but different requirements. All of the messages were grouped by sending process, and when the sending process said its session was over its messages were fed into a post-processing queue. There was support for holding messages in memory or on disk, and data-preserving fallbacks if the post-processing halted or got too backed up. The message rate varied quite a bit; during the US business day the rate was at least doubled. We used that to give the post-processing queue time to catch up off-hours.
This was designed about 16 years ago, so well before the big-data tools, ssds, or powerful machines we have today.