Re:Is Your Linux Version Hiding Interrupt CPU Usage from You?
by andy_threos_io on Monday February 15, 2021 @10:46AM
(#26143780)
Irq accounting depends on architecture and code usage, e.g. like some kind of barrier (serializing) before the RDTSC.
on newer x86_64 the accounting code is around 100-200 cycles, so not a big deal, interrupt return and exit code is more than that (save context, load new context etc.).
The RDTSC and fence timings (Latency, Reciprocal throughput) [0]:
Re:Is Your Linux Version Hiding Interrupt CPU Usage from You?
by tanelpoder on Monday February 15, 2021 @10:58AM
(#26143906)
I guess the fence instructions are there for avoiding the CPU out-of-order execution getting the timestamp too early (or late) compared to the interrupt service routine's start & end?
Re:Is Your Linux Version Hiding Interrupt CPU Usage from You?
by krzyk on Monday February 15, 2021 @04:01AM
(#26140681)
CONFIG_IRQ_TIME_ACCOUNTING according to the docs:
CONFIG_IRQ_TIME_ACCOUNTING:
Select this option to enable fine granularity task irq time
accounting. This is done by reading a timestamp on each
transitions between softirq and hardirq state, so there can be a
small *performance impact*.
If in doubt, say N here.
So I didn't enable this option in my kernels afraid of loosing performance on my laptop.
Re:Is Your Linux Version Hiding Interrupt CPU Usage from You?
by tanelpoder on Monday February 15, 2021 @11:02AM
(#26143942)
(Author here). It would be interesting to test the performance difference. I downloaded the source of one of my distro-provided kernels & recompiled it with different config setting (to compare perf with just one thing changed), but got into some trouble with my drivers and ran out of time.
I would say that on your laptop, you probably wouldn't even notice a difference (as the "rdtsc" timestamp reading operations are pretty efficient & done entirely within a CPU). But with anything that can interact with external hardware million(s) of times per second (heavy SSD I/O in servers and moving large amounts of data over network), may see a minor (but measurable) hit.
In my experience, when large amounts of CPU gets used by interrupt routines (either hardware or software interrupts), that more likely indicates some bug or configuration issue in a device driver or even some faulty hardware issue.
edit: so, running "dmesg" and checking hardware health metrics (mceloc, EDAC) is a reasonable thing to do when interrupt storms & related CPU usage suddenly show up.
Re:Is Your Linux Version Hiding Interrupt CPU Usage from You?
by londons_explore on Monday February 15, 2021 @03:50AM
(#26140583)
Does this CPU time accounting have any noticeable performance impact?
I know in some embedded usecases, there are hundreds of thousands or millions of interrupts per second, and an interrupt routine might be only tens of clock cycles long. If that's the case, an extra bunch of instructions to time how long every single interrupt takes is a substantial overhead.
Re:Is Your Linux Version Hiding Interrupt CPU Usage from You?
by shakna on Monday February 15, 2021 @05:19AM
(#26141218)
> Does this CPU time accounting have any noticeable performance impact?
Yes.
> Select this option to enable fine granularity task irq time accounting. This is done by reading a timestamp on each transitions between softirq and hardirq state, so there can be a small performance impact.
Re:Is Your Linux Version Hiding Interrupt CPU Usage from You?
by archi42 on Monday February 15, 2021 @08:30AM
(#26142524)
Ahh, yes, that's actually a great point to be aware of, thanks @tanelpoder.
Looking at my machine: Funny to see 2h of interrupt processing over 12d uptime. Not that this is bad (0.80% on avg), but my first computer would have taken >6 weeks to process these interrupts - and not doing anything else. And that's only going by the clocks, a 80386 has a much worse IPC/pipeline than a 6th gen i7.
Re:Is Your Linux Version Hiding Interrupt CPU Usage from You?
by 3np on Monday February 15, 2021 @11:54AM
(#26144443)
Thank you, this is something I somewhat recently realized I don't know how to measure and troubleshoot properly - I've never been able to give a confident answer to if context switching/interrupts are a bottleneck or not, just "the number looks highish and I don't have anything else to blame".
Re:Is Your Linux Version Hiding Interrupt CPU Usage from You?
by joosters on Monday February 15, 2021 @05:29AM
(#26141285)
The tool 'powertop' is very useful here, it will show what is causing any form of wakeup on your system, and offers a 'Tunables' tab with suggested settings to improve some aspects.
Re:Is Your Linux Version Hiding Interrupt CPU Usage from You?
by LinuxBender on Tuesday February 16, 2021 @01:23PM
(#26157023)
Powertop is great. You can even run it in non-interactive mode. That said, it only optimizes for lower power usage and lower interrupts. You can further reduce interrupts by disabling devices. If there are devices you don't need, you can blacklist them in /etc/modprobe.d/ provided you understand the potential dependencies other devices may have on the drivers. This can free up a tiny bit of memory as well.
on newer x86_64 the accounting code is around 100-200 cycles, so not a big deal, interrupt return and exit code is more than that (save context, load new context etc.).
The RDTSC and fence timings (Latency, Reciprocal throughput) [0]:
[0] https://www.agner.org/optimize/instruction_tables.pdfI would say that on your laptop, you probably wouldn't even notice a difference (as the "rdtsc" timestamp reading operations are pretty efficient & done entirely within a CPU). But with anything that can interact with external hardware million(s) of times per second (heavy SSD I/O in servers and moving large amounts of data over network), may see a minor (but measurable) hit.
In my experience, when large amounts of CPU gets used by interrupt routines (either hardware or software interrupts), that more likely indicates some bug or configuration issue in a device driver or even some faulty hardware issue.
edit: so, running "dmesg" and checking hardware health metrics (mceloc, EDAC) is a reasonable thing to do when interrupt storms & related CPU usage suddenly show up.
I know in some embedded usecases, there are hundreds of thousands or millions of interrupts per second, and an interrupt routine might be only tens of clock cycles long. If that's the case, an extra bunch of instructions to time how long every single interrupt takes is a substantial overhead.
Yes.
> Select this option to enable fine granularity task irq time accounting. This is done by reading a timestamp on each transitions between softirq and hardirq state, so there can be a small performance impact.
[0] https://cateee.net/lkddb/web-lkddb/IRQ_TIME_ACCOUNTING.html
Looking at my machine: Funny to see 2h of interrupt processing over 12d uptime. Not that this is bad (0.80% on avg), but my first computer would have taken >6 weeks to process these interrupts - and not doing anything else. And that's only going by the clocks, a 80386 has a much worse IPC/pipeline than a 6th gen i7.