News for hackers. Stuff that matters.
Re:The Problem with ‘5 Whys’ (2016) by jimbokun on Wednesday April 10, 2019 @01:21PM (#19626820)
I don't understand, if the bug was written by your team, how could the ultimate cause be something outside of your team?

Seems like 5 Whys in that scenario would lead you to look at things like insufficient test coverage, no code review, code reviewers missed the bug because...

Those paths of inquiry lead to your team changing things under your direct control that could make it less likely for similar bugs to reach production in the future.

Re:The Problem with ‘5 Whys’ (2016) by espadrine on Wednesday April 10, 2019 @01:59PM (#19627196)
Broken expectations are a cause.

For instance, when ext4 gained traction, some users found a number of zeroed-out files after a crash.

Why? Because the programs managing those files updated them by truncating them and then writing them.

Why? Because fopen() has no "replace file" mode, even though the "w" mode seems to do just that as long as the system doesn't crash (it does O_WRONLY|O_CREAT|O_TRUNC).

Why? Because file systems used to not have delayed allocation when fopen() was created. ext4 introduced the concept to the ext family.

Why? Because ext4 was designed to improve performance compared to ext3, and that change does that without breaking POSIX.

Why? Because, even with ext3, the only way to correctly implement "replace file" on POSIX is to write the replacement file on the same directory, fsync() the file, close the file descriptor, fsync() the directory, and rename the file.

At that point, the whys have shifted the blame through userland programmers, POSIX fopen(), ext4, and back to userland programmers. Depending on where you stop, the solution can be:

1. fix all userland programs,

2. change ext4 to force sync upon closing a truncated file's descriptor (which got implemented),

3. upgrade the POSIX standard to have a dedicated "replace file" function call.

The truth is, userland is buggy, and the POSIX standard is arcane, so they both share blame. (But ext4, which was correctly implemented, is the only thing that got patched.)

http://dream.thunk.org/tytso/blog/2009/03/12/delayed-allocat...

Re:The Problem with ‘5 Whys’ (2016) by grok2 on Thursday April 11, 2019 @12:56AM (#19631271)
To me it seems like ext4 was the problem here -- it tried to provide a feature that wasn't adequately supported by the underlying OS.
Re:The Problem with ‘5 Whys’ (2016) by espadrine on Thursday April 11, 2019 @05:09AM (#19632296)
Delayed allocation was adequately supported.

Implementing it in ext4 increased the risk of zeroing out a file from “having all files updated in the past 20ms be zeroed out” to “having all files updated in the past 4 seconds be zeroed out” in the event of a system crash.

Re:The Problem with ‘5 Whys’ (2016) by tedunangst on Wednesday April 10, 2019 @01:41PM (#19627025)
Team two promised us this service would never fail, so we didn't check for errors...
Re:The Problem with ‘5 Whys’ (2016) by ben509 on Wednesday April 10, 2019 @06:03PM (#19629220)
The root cause may be something outside your team, but that doesn't mean you're saying you'll sit on your hands. "The service we depend on failed" can still mean, "and we need to implement better retries or redundancy" as a remedy.