Yup, been there too, way, way back.

We wrote simulation tools for typical access patterns and ran them heavily for testing out new cards, both for performance and failure rate. Luckily there was enough margin on the devices that we (R&D) could select the more expensive industrial graded cards by good manufacturers, but I did get to see some really shitty cards that purchasing preferred (because they had gotten a great price on them).

However in our case the corruption was due to a buggy FAT-driver for the obscure RTOS we used. In the end though I learned a lot about FAT which was fun (<sarcasm>and is a highly marketable skill these days</sarcasm>).

Haha, I ended up tapping all read and writes to the card, intending to catch our crappy FAT driver in the act of corrupting.

Turns out the card itself was shit and replaying the trace would just kill arbitrary cards.

Haha, I spent so much time debugging their crappy FAT-driver. Another time the whole system would just hang when you created a new file in a directory that already contained some files. Turns out that their long-to-short filename algo was:

For the first 4 files with identical prefix use the ~N scheme as in LongFile.txt -> LONGFI~1.TXT For the following files it was: 1) "LONG" + hex_str(hash(long_file_name)) + ".TXT" 2) if there's a name collision repeat step 1.

Compound that with a the fact that hash() was implemented something like:

    int h = 0;
    for(int i = 0; i < strlen(long_file_name); ++i)
    {
      h = (h + long_file_name[i]) ^ 0x42424242;
    }

It's pretty easy to see where this falls apart...

How does one tap all the reads and writes to a card?

Just intercept the read and write calls! On linux you can just use 'strace $prog'; try it on a minimsal program that reads and writes a file. For a custom approach, this can be done with a library (using LD_PRELOAD). Plenty of articles about how to do this, e.g. http://www.linuxjournal.com/node/7795 (essentially the same technique on Windows, or other OS with shared libs). Slightly more hardcore is to hook the kernel syscalls; not normally necessary (on linux this is effectively hooking the other side of glibc, where it talks to the kernel) but if all you've got a static binary it's one approach.

Note that all these techniques only show you the read and write calls made to the library/OS and - importantly - not what actually happens to the card. To see that, the next level down is to instrument the card driver to track the actual I/O operations (i.e. so you see what the card is really being asked to do, sans all the caching and buffering).

Note that's not the end of the story; there's what the hardware controller decides to do and when the hardware actually reads/writes the flash array. That's the level where the quality of the firmware in the controller(s) matters.

I suspected the filesystem driver itself, so I had to modify the driver rather than strace (also this was WinCE quite unfortunately).

Doesn't that mean you then have to write an event to disk for every write-to-disk event?

I would log by RPC or whatever to a remote computer.

Modify the SD card driver to open a socket to a computer in the lab, and just log it's commands and responses over TCP with timestamps. The network was orders of magnitude faster than the crappy SDIO interface, so it was easy this time. I've had to do a similar thing with SAS, and you basically just have to pay tens of thousands for a special purpose protocol analyzer, or build your own with an FPGA.

And write what you need to write to disk using an FEC so you can read it back off when it is corrupt. Effectively do what the card should be doing in the first place.

Image the system, and confirm the checksums on first initial boot, then never re-write any of them.

Sounds like a good description of all those "It runs Linux because here is the kernel error" images that bounce around the web. Most of them from signage solutions or similar that stay on for ages at a time, and invariably comes down to the storage device going belly up.

this is the story of my life (I run a signage business).

The rpi2 seemed to be the worst for fs corruption, I've never had a pi1 fail on me. Jury is still out on the pi3.

Have a similar situation. Would be helpful (and grateful) if you would expand on what you learned to minimize writes.