>It definitely made me wonder how many web servers out there have their performance determined by the rate at which logs can be written to disk.

As someone in the HFT space I suspect a lot of traders are performance bound by this. Wouldn't surprise me at all if a couple darkpools were this way.

As someone formerly in that space I don't think so, assuming by "traders" you mean the prop firms who actually need the speed advantage. You can always arrange for the logging to happen outside the critical path, which is a tiny minority of the code you run. And you will do that if its necessary to gain a fraction of a microsecond on a competitive trade. Even if it's just being able to run a build with logging off for special cases.

For a looser definition of "traders" that includes dark pool operators and exchanges, then sure. Writing logs was definitely the bottleneck at one major European exchange for some years. They may not be able to turn off logging for regulatory reasons, but they also don't care so much about the microseconds. In this case they could definitely have improved performance with horizontal scaling (have each computer handle fewer stock symbols) but didn't consider that economically justified.

By traders I mean "anyone who doesn't sit on the market side of my network card", so a wide variety of firms. My comment about darkpools was tangential but there's a couple of them who aren't exactly known for running a tight ship.

>You can always arrange for the logging to happen outside the critical path, which is a tiny minority of the code you run. And you will do that if its necessary to gain a fraction of a microsecond on a competitive trade. Even if it's just being able to run a build with logging off for special cases.

Yes you can do that. Yes you'd think that anyone for whom it would be beneficial would do that... but you'd be surprised how often people run with logs cranked to 11 in prod.

I've had the opposite. Where we couldn't turn logging off because the writes prevented a weird race condition that would crash the application.

Lmao why use data synchronization when you can just write to disk?

Heh. It was a bit hard to pin down, but we did eventually fix it. One of those Java super deep dependency things where the 3rd party thing at the bottom is broken. I think it was cxf.

Why write to disk when you can just sleep() sporadically?

Trading systems often record every single packet going through the network using fiber optic splitters, which makes it pretty easy to replay traffic to a debugging instance.

Most teams don't understand the cost of logging on their infrastructure, they just seem to thin writing logs is 'free'.