The exact same silent data corruption issues just happened to my 6 x 5TB ZFS FreeBSD fileserver. But unlike what the poster concluded, mine were caused by bad (ECC!) RAM. I kept meticulous notes, so here is my story...

I scrub on a weekly basis. One day ZFS started reporting silent errors on disk ada3, just 4kB:

    pool: tank
   state: ONLINE
  status: One or more devices has experienced an unrecoverable error.  An
          attempt was made to correct the error.  Applications are unaffected.
  action: Determine if the device needs to be replaced, and clear the errors
          using 'zpool clear' or replace the device with 'zpool replace'.
     see: http://illumos.org/msg/ZFS-8000-9P
    scan: scrub repaired 4K in 21h05m with 0 errors on Mon Aug 29 20:52:45 2016
  config:
          NAME        STATE     READ WRITE CKSUM
          tank        ONLINE       0     0     0
            raidz2-0  ONLINE       0     0     0
              ada3    ONLINE       0     0     2  <---
              ada4    ONLINE       0     0     0
              ada6    ONLINE       0     0     0
              ada1    ONLINE       0     0     0
              ada2    ONLINE       0     0     0
              ada5    ONLINE       0     0     0

I monitored the situation. But every week, subsequent scrubs would continue to find errors on ada3, and on more data (100-5000kB):

  2016-09-05: 1.7MB silently corrupted on ada3 (ST5000DM000-1FK178)
  2016-09-12: 5.2MB silently corrupted on ada3 (ST5000DM000-1FK178)
  2016-09-19: 300kB silently corrupted on ada3 (ST5000DM000-1FK178)
  2016-09-26: 1.8MB silently corrupted on ada3 (ST5000DM000-1FK178)
  2016-10-03: 3.1MB silently corrupted on ada3 (ST5000DM000-1FK178)
  2016-10-10: 84kB silently corrupted on ada3 (ST5000DM000-1FK178)
  2016-10-17: 204kB silently corrupted on ada3 (ST5000DM000-1FK178)
  2016-10-24: 388kB silently corrupted on ada3 (ST5000DM000-1FK178)
  2016-11-07: 3.9MB silently corrupted on ada3 (ST5000DM000-1FK178)

The next week. The server became unreachable during a scrub. I attempted to access the console over IPMI but it just showed a blank screen and was unresponsive. I rebooted it.

The next week the server again became unreachable during a scrub. I could access the console over IPMI but the network seemed non-working even though the link was up. I checked the IPMI event logs and saw multiple correctable memory ECC errors:

  Correctable Memory ECC @ DIMM1A(CPU1) - Asserted

The kernel logs reported muliple Machine Check Architecture errors:

  MCA: Bank 4, Status 0xdc00400080080813
  MCA: Global Cap 0x0000000000000106, Status 0x0000000000000000
  MCA: Vendor "AuthenticAMD", ID 0x100f80, APIC ID 0
  MCA: CPU 0 COR OVER BUSLG Source RD Memory
  MCA: Address 0x5462930
  MCA: Misc 0xe00c0f2b01000000

At this point I could not even reboot remotely the server via IPMI. Also, I theorized that in addition to correctable memory ECC errors, maybe the DIMM experienced uncorrectable/undetected ones that were really messing up the OS but also IPMI. So I physically removed the module in "DIMM1A", and the server has been working perfectly well since then.

The reason these memory errors always happened on ada3 is not because of a bad drive or bad cables, but likely due to the way FreeBSD allocates buffer memory to cache drive data: the data for ada3 was probably located right on defective physical memory page(s), and the kernel never moves that data around. So it's always ada3 data that seems corrupted.

PS: the really nice combinatorial property of raidz2 with 6 drives is that when silent corruption occurs, the kernel has 15 different ways to attempt to rebuild the data ("6 choose 4 = 15").