Re:War story: the hardest bug I ever debugged
by wasabiketchup on Thursday March 27, 2025 @10:15AM
(#43493904)
Fun one:
I work on a server software of online backups for customers. We do daily thousands of mount/umount of a particular filesystem.
Once every month or so, we get an issue where a file timestamp fails to save, the error happens at the filesystem level.
Hard to reproduce! It's a filesystem bug! So it's full theorical, reading code and seeing how it would happen.
Found out after a while, the conditions were fun. I don't remember exactly, but it was like, you need to follow these steps :
1/ Create a folder
2/ Create in it 99 files (no more no less)
3/ Create a new folder
4/ Copy the first of the 99 files in the new folder
The issue was linked to some data structure caching, and cache eviction.
I work on a server software of online backups for customers. We do daily thousands of mount/umount of a particular filesystem. Once every month or so, we get an issue where a file timestamp fails to save, the error happens at the filesystem level.
Hard to reproduce! It's a filesystem bug! So it's full theorical, reading code and seeing how it would happen.
Found out after a while, the conditions were fun. I don't remember exactly, but it was like, you need to follow these steps : 1/ Create a folder 2/ Create in it 99 files (no more no less) 3/ Create a new folder 4/ Copy the first of the 99 files in the new folder
The issue was linked to some data structure caching, and cache eviction.
Had fun finding it out!