This is my experience as well: almost no existing software is written with ultra low latency in mind (it's optimized for throughput instead).

Getting latency in the micros with low jitter is extremely difficult and requires a bunch of custom software. And getting into the nanos requires a bunch of custom hardware.