That sounds like an interesting detective story. Did they write it up anywhere?
Core dumps didn’t show anything useful because the crashes were just that a pointer was mysteriously pointing into a bad place. But how did it get corrupted?
We were doing increasingly insane things to reproduce the problem locally.
An interesting idea I had to reproduce it was using a database replica, which did crash sometimes.
We used TCPDump to capture the replication stream, then waited a week for the replica to crash. We could start up a copy of the replica in the state it was at the beginning and replay the week of replication data into the socket to reproduce the problem.
My theory was that since a DB replica doesn’t do anything other than the sequence of operations that come to it via the replication stream, that this should deterministicly reproduce the problem.
Turns out that it didn’t.
Anyway, if my memory is correct, the issue was that memcpy was copying an extra 1-3 bytes at the end of the range, if the start of the range was unaligned. If there happened to be a pointer there then that would lead to a crash down the line. My memory could be off though.
Ahhh I could see how that could be! Fascinating. That's exactly the kind of heisenbug I used to love. I always liked the ones that seemed literally impossible but had a nice explanation.
That is surprising outside of the kernel. There are a large number of programs that would fail very hard very fast with that sort of artificial restriction.
We've become so accustomed to computers being generally unreliable that I guess people see something weird and just reboot. We checksum, run reductant servers, and just generally tolerate way more failures and weirdness than we should.
It's funny because we're theoretically an industry that could be full of maths and proofs but ask anybody outside of it and you'll get "yeah computers just don't work sometimes I guess that's how it is"
And once you start designing around that unreliability, it is harder to justify chasing esoteric bugs rather than folding that alongside the probability model of random hardware failure.
It’s hard for any system dealing with the real world to be completely “maths and proofs”. There is plenty of it that’s applied even in the real world systems we operate today. At the end of the day, it’s Engineering and not a pure science.
And I guess the other problem is that quite often you don’t need to quantify the failure modes and do “proper engineering”, because a lot of the systems we build and use just don’t need the level of reliability that e.g. a bridge needs (yet). There are definitely industries where software reliability is considered paramount, but that’s not the majority of the market.
Sure. As the saying goes, "Anybody can build a bridge that never falls down. But it takes an engineer to build a bridge that _almost_ doesn't fall down" (referring to being able to build it in a cost-conscious way). There's a spectrum there for sure, but as an industry we're on the other side of it from where I personally wish we were.
memmove was broken on 32-bit executables on systems with SSE2 whenever the move occurred over the 2GB memory boundary due to an issue with signed vs unsigned integers.
The stacktrace said the crash came from SSE memcpy. I did not bother investigating it, since I have no users anyways.
I had an really old CPU (like i5-520m). Perhaps the code required some SSE feature, it did not have
That was libc. I also use Pascal, which has its own memcpy implementation. A while ago, they discovered it does not handle alignment correctly. (like it would adjust by 4-i when it should have adjusted by i) I thought that is what I get when using Pascal rather than a more commonly used language, but apparently nothing works
We had an issue for years where occasionally our backend database would fall over and we were just seeing strange corruption in the stored data sometimes.
Turns out that memcpy() didn’t handle data that was not aligned to a 4 byte boundary correctly in the 32 on 64 case. It took one of our sysadmins a very very long time to find it.
What blew me away is the idea that there can be a bug in something that is used literally everywhere all the time.
In addition, there was a pull request with a fix that someone had written that got ignored for years that we found. We just applied the diff in the pull request and it fixed our issues. The fix was eventually applied upstream a few years later.