I've had the same thing happen on a similar system, with ECC. In that case I eventually tracked it down to a dodgy power supply, but the damage was done... it was time to dump and reload.

Corruption happens. A good filesystem needs to cope.

The memory is ECC. No data I care about was affected. I'm just surprised there's no "do whatever you need to make it consistent so I can move on with life" option and instead I have to bring up a whole new pool.

You should post on the illumos zfs-discuss mailing list. The most amount of experts will be there to provide comments. I just saw the hostname of the machine; your "clout" may get you some attention. :-)

You very well may have some flaky hardware which is causing the issue, in which case, it probably needs to be fixed sooner than later. And maybe with some magic, you can get the pool into a happy state without rebuilding.

If you are having random errors like that it might be a sign of something bigger.

As mentioned in my other response I had random bit flips in memory. After I replaced my broken NIC crashes stopped. But I got a nasty surprise when I removed a filesystem - kernel panic due to failed assertion, and then kernel panic for the same assertion every single time I tried to import the pool.

I suspect it possibly could be resolved somehow without losing data, but decided to simply restore it since my backup was relatively fresh.

I suspect you might have something similar to what I experienced, and even though you use ECC memory I doubt it would help much when corruption is happening on a bus.