Re:Linux: IO without entering the kernel
by rwmj on Sunday December 16, 2018 @08:15AM
(#18692925)
It's not at all unusual for our HFT and other realtime customers to dedicate cores to particular tasks. They often boot with isolcpus to reduce the number of CPUs visible to the scheduler, then dedicate those CPUs to servicing interrupts or running critical programs. They also turn off all power management so the CPUs are burning at 100% all the time.
[ Parent
| on HN ]
Re:Linux: IO without entering the kernel
by CoolGuySteve on Sunday December 16, 2018 @12:51PM
(#18694143)
Standardizing the kernel-bypass of all the different network cards for absolute minimal latency would make life much easier in HFT.
> They also turn off all power management so the CPUs are burning at 100% all the time.
That seems quite silly to me. That wastes a bunch of TDP and likely loses some Turbo.
Re:Linux: IO without entering the kernel
by bdavis__ on Sunday December 16, 2018 @03:19PM
(#18694972)
you do this to minimize latency. to ensure things happen in usecs instead of msecs, this is what you have to do.
deliberate tradeoff. it is better to waste electricity, and to ensure your deadlines are met than for some to be late and some to be early.
changing the speed of a cpu is very expensive in time, and can take a couple of ms to complete (of course, most of this is caused by how it is mechanized in the OS).
Transitions from C0 to C1 and back are very fast [0], and basically any application has at least a core and probably several doing kernel things like servicing network IRQs.
I agree that HFT applications should not be using deep C states.
[0] Unless you have a misguided feature like C1E auto-promotion on. Fortunately, recent Linux kernels will overriding firmware and turn it off. On Sandy Bridge and possibly newer CPUs with C1E auto-promotion on, resuming from C1 can take many, many milliseconds.