News for hackers. Stuff that matters.
Re:Fail-slow at scale: Evidence of hardware performance degr... by gpapilion on Monday February 26, 2018 @06:51AM (#16464657)
I had a large scale issue caused by a bug in the fan control system reading a temp that was deadlocked. This caused cpu performance to tank, since the cpu was thermal throttled.

To make matters worse the environmental sensors reported no issues, because they were all frozen at a single reading as well.

Re:Fail-slow at scale: Evidence of hardware performance degr... by LeonM on Monday February 26, 2018 @06:10AM (#16464517)
> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance.

I've never heard of such thing, can vibrations from a fan really influence disk performance?

Re:Fail-slow at scale: Evidence of hardware performance degr... by blattimwind on Monday February 26, 2018 @06:11AM (#16464520)
Yes. Loud noises and even yelling degrade disk performance as well.

https://www.youtube.com/watch?v=tDacjrSCeq4

Re:Fail-slow at scale: Evidence of hardware performance degr... by joncrane on Monday February 26, 2018 @08:41AM (#16465150)
This is one of the classics! I love it. Screaming at a HDD.
Re:Fail-slow at scale: Evidence of hardware performance degr... by godelmachine on Monday February 26, 2018 @08:49AM (#16465192)
I started screaming in my cubicle watching this.
Re:Fail-slow at scale: Evidence of hardware performance degr... by meuk on Monday February 26, 2018 @08:13AM (#16465001)
In the days of burning CD's I was always told that you should be quiet and not touch the PC when you were burning. In my experience there was indeed a correlation between noise and the success of burning. I'm not sure if this is a placebo effect or not, and if this also holds for reading CD's.

On the other hand, if you have a laptop with an SSD (which is non-mechanical) there shouldn't be too much sensitivity to vibrations, I think.

Re:Fail-slow at scale: Evidence of hardware performance degr... by janci on Monday February 26, 2018 @10:43AM (#16466039)
I remember how I didn't dare to touch the computer during burning other than move the mouse a few pixels here and there so it has it peace and the screensaver do not kick in. CDs were expensive and burnig was a hour-long process.
Re:Fail-slow at scale: Evidence of hardware performance degr... by jebeng on Monday February 26, 2018 @03:22PM (#16468624)
That upgrade path from 2x to 52x was so nice though.
Re:Fail-slow at scale: Evidence of hardware performance degr... by gpapilion on Monday February 26, 2018 @12:46PM (#16467238)
I previously had a vendor replace hundreds of systems due to a poor fan design where they mounted it under a spinning disk. It is very surprising how sensitive they are.
Re:Fail-slow at scale: Evidence of hardware performance degr... by vardump on Monday February 26, 2018 @06:18AM (#16464551)
Yes. Mechanical disks are very sensitive to vibrations.
Re:Fail-slow at scale: Evidence of hardware performance degr... by godelmachine on Monday February 26, 2018 @07:04AM (#16464707)
Blue screen errors are most often caused by RAM issues. I remember very well that I was facing blue screen error at my work once, & upon searching I found that all I need to do is remove the RAM, clean and dust it, and put it back into the same or another RAM socket.
Re:Fail-slow at scale: Evidence of hardware performance degr... by xkgt on Tuesday February 27, 2018 @09:47AM (#16473730)
This is a HN gem! Wonder how it didn't make it to first page.

I once had a mobile app perform very poorly only for large downloads on a particular SSID. No other applications had such issue. It turned out to be due to a firmware bug which scrapped TCP SACK field, but the problem manifested only for large data transfers over unreliable, high latency connections. Got us all chasing our tails for a while though...

Re:Fail-slow at scale: Evidence of hardware performance degr... by riza_on on Thursday March 01, 2018 @03:36PM (#16496015)
The authors are still collecting more stories for their longer journal version of the paper. If your institution has 10+ stories to contribute and are interested to be part of the journal paper, please go ahead contact the first two authors of the original paper.