Haha, I ended up tapping all read and writes to the card, intending to catch our crappy FAT driver in the act of corrupting.

Turns out the card itself was shit and replaying the trace would just kill arbitrary cards.

Haha, I spent so much time debugging their crappy FAT-driver. Another time the whole system would just hang when you created a new file in a directory that already contained some files. Turns out that their long-to-short filename algo was:

For the first 4 files with identical prefix use the ~N scheme as in LongFile.txt -> LONGFI~1.TXT For the following files it was: 1) "LONG" + hex_str(hash(long_file_name)) + ".TXT" 2) if there's a name collision repeat step 1.

Compound that with a the fact that hash() was implemented something like:

    int h = 0;
    for(int i = 0; i < strlen(long_file_name); ++i)
    {
      h = (h + long_file_name[i]) ^ 0x42424242;
    }

It's pretty easy to see where this falls apart...

How does one tap all the reads and writes to a card?

Just intercept the read and write calls! On linux you can just use 'strace $prog'; try it on a minimsal program that reads and writes a file. For a custom approach, this can be done with a library (using LD_PRELOAD). Plenty of articles about how to do this, e.g. http://www.linuxjournal.com/node/7795 (essentially the same technique on Windows, or other OS with shared libs). Slightly more hardcore is to hook the kernel syscalls; not normally necessary (on linux this is effectively hooking the other side of glibc, where it talks to the kernel) but if all you've got a static binary it's one approach.

Note that all these techniques only show you the read and write calls made to the library/OS and - importantly - not what actually happens to the card. To see that, the next level down is to instrument the card driver to track the actual I/O operations (i.e. so you see what the card is really being asked to do, sans all the caching and buffering).

Note that's not the end of the story; there's what the hardware controller decides to do and when the hardware actually reads/writes the flash array. That's the level where the quality of the firmware in the controller(s) matters.

I suspected the filesystem driver itself, so I had to modify the driver rather than strace (also this was WinCE quite unfortunately).

Doesn't that mean you then have to write an event to disk for every write-to-disk event?

I would log by RPC or whatever to a remote computer.

Modify the SD card driver to open a socket to a computer in the lab, and just log it's commands and responses over TCP with timestamps. The network was orders of magnitude faster than the crappy SDIO interface, so it was easy this time. I've had to do a similar thing with SAS, and you basically just have to pay tens of thousands for a special purpose protocol analyzer, or build your own with an FPGA.