It's also tricky with a COW filesystem like ZFS in which the order of data on-disk may not match the logical order of the file. So your database may decide to dispatch a bunch of sequential 1MB reads to the filesystem, only for the filesystem to translate those large sequential reads into many small random 8kB reads. It depends on how fragmented the file is, so you can start off with great performance which degrades significantly over time.

Luckily SSDs have a much lower penalty for random reads compared to spinning disks, so in future this will be less of an issue.

In the meantime all you can do is use a different filesystem, hide the problem with cache, or manually defragment your data files from time to time.