The real culprit here is really "fsync is slow". A couple of months ago I write a little test that basically wrote a 1MB file, in increments of 16 byte write() calls, followed by either fsync() fdatasync() or.. nothing. That is, basically:

  while(written < 1MB) {
     written += write(fd,buf,chunksz);
     //fsync(fd);
     //fdatasync(fd);
  }

Here's some of the times it took to write a 1MB file (Fedora 14, old hard drive)

  chunksz = 16:
  No sync'ing: 926ms
  fdatasync: 727114ms
  fsync: 3024498ms
  (yes, THAT slow)

  chunksz = 256:
  No sync'ing: 65ms
  fdatasync: 53786ms
  fsync: 191553ms

  chunksz=1024:
  No sync'ing: 20ms
  fdatasync: 20232ms
  fsync: 48039ms