News for hackers. Stuff that matters.
Re:Deep inside the Linux kernel: a network latency spike by mey on Thursday November 19, 2015 @06:46PM (#10598537)
Did I misread the article or did they actual prove that the tcp_collapse was the root cause of their 1000ms spike?
Re:Deep inside the Linux kernel: a network latency spike by majke on Thursday November 19, 2015 @06:50PM (#10598562)
The working theory is that "tcp_collapse" causes spikes of ~20 ms, which in turn make Linux not receive packets in time, which makes the network buffers to fill, which means new packets will be lost, which may be visible as 1s... latency spike end-to-end.

This is for the TCP case. For the ICMP, indeed, the 1.8s times are not explained, but it's totally possible that 100 of 18ms events accumulated. Remember that the ping was running for 32 hours, while the system tap scripts for 30 seconds.

Re:Deep inside the Linux kernel: a network latency spike by gricardo99 on Thursday November 19, 2015 @07:21PM (#10598718)
Not specifically tcp_collapse, but the article theorizes that some memory management done by tcp_collapse (i.e. GC) is the cause. I'd place my bets on memory calls being the cause too.

But... they never went to the next step and confirmed that tcp_collapse was making lengthy system memory calls. They kind of gloss over the details of why reducing rmem alleviates the problem.

Is it because with smaller rmem tcp_collapse aggregates segments into smaller sk_buffs, and these smaller-size memory calls don't take as long to return?

Or is it because tcp_collapse has less memory "stitching" to do, since it can't fit as many segments into the smaller sk_buffs?

Re:Deep inside the Linux kernel: a network latency spike by btilly on Thursday November 19, 2015 @07:32PM (#10598762)
My understanding of the article was that the amount of data that they could be trying to collapse had been up to 32 MB, and now was never more than 2 MB.

Doing a global operation on a large data structure takes longer than doing a global operation on a small data structure. Average throughput may be better if you do that, but maximum latency was the issue here.

Re:Deep inside the Linux kernel: a network latency spike by acqq on Friday November 20, 2015 @05:53AM (#10600622)
Thanks, I missed the detail that their maximum buffer size was 32 MB. They speak about the default size but then change the maximum size and show that the longer delays don't happen.
Re:Deep inside the Linux kernel: a network latency spike by abrookewood on Thursday November 19, 2015 @07:30PM (#10598750)
I read it as a smaller buffer meant that when garbage collection kicked in, it was working on a smaller amount of memory and was therefore faster to complete.