71 TiB DIY NAS Based on ZFS on Linux

Re:71 TiB DIY NAS Based on ZFS on Linux by kogir on Sunday August 31, 2014 @07:00PM (#8250729)

ZFS is pretty great, but it's not perfect. Occasionally you have to dump and re-create a whole pool to fix corruption:

    [[email protected] ~]$ zpool status -v
      pool: arc
     state: ONLINE
      scan: scrub repaired 0 in 0h43m with 0 errors on Wed Aug 13 20:14:53 2014
    config:
    
        NAME          STATE     READ WRITE CKSUM
        arc           ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            gpt/arc0  ONLINE       0     0     0
            gpt/arc1  ONLINE       0     0     0
    
    errors: No known data errors
    
      pool: tank
     state: ONLINE
    status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
    action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
       see: http://illumos.org/msg/ZFS-8000-8A
      scan: scrub repaired 0 in 5h3m with 0 errors on Tue Aug 26 00:19:05 2014
    config:
    
        NAME           STATE     READ WRITE CKSUM
        tank           ONLINE       0     0     0
          raidz2-0     ONLINE       0     0     0
            gpt/tank0  ONLINE       0     0     0
            gpt/tank1  ONLINE       0     0     0
            gpt/tank2  ONLINE       0     0     0
            gpt/tank3  ONLINE       0     0     0
        logs
          mirror-1     ONLINE       0     0     0
            aacd5p1    ONLINE       0     0     0
            aacd6p1    ONLINE       0     0     0
    
    errors: Permanent errors have been detected in the following files:
    
            /usr/nginx/var/log/nginx-access-appengine.log.0

I can't delete that file or recover it from backup, and don't really trust the pool or our disk controller anymore. So we're moving to a new server, again. Good times.

I imagine this would be disastrous for someone who's filled a 24 disk array with data, and has to purchase a duplicate to safely restore to.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by laumars on Monday September 01, 2014 @04:03AM (#8251926)

You don't need to drop the pool to clear those error messages:

    zpool clear tank

Source: http://docs.oracle.com/cd/E19253-01/816-5166/6mbb1kqod/index... (unfortunately there's no tag ID's to hash to, so you'll have to [ctrl]+[F] to find the specific section on 'clear')

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by kogir on Monday September 01, 2014 @09:52PM (#8255450)

Sadly, that only clears device errors. Tried it with no change.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by laumars on Tuesday September 02, 2014 @02:09AM (#8256038)

Dropping the pool and recreating it only clears the errors as well.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by takeda on Monday September 01, 2014 @03:46AM (#8251880)

I got ZFS corruption myself. Although another issue (which actually brought my attention) was also common system crashes, whenever ARC was enabled. When inspecting cored dump it looked like random bit flips.

In my case the problem turned out to be a broken WiFi card (PCI). After I replaced it no issues for 2 years now.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by dsturnbull2049 on Sunday August 31, 2014 @07:48PM (#8250905)

This is one of the key strengths of ZFS. Your log file would be silently corrupted without checksumming.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by rodgerd on Sunday August 31, 2014 @08:49PM (#8251049)

Regardless, it's still frustrating that so many failure modes for ZFS have an answer of "blow it all away and start again". And that's with Solaris/ZFS.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by laumars on Monday September 01, 2014 @04:09AM (#8251938)

There are other ways to fix issues with ZFS pools without having to "blow it all away and start again". In the example that started this conversation fork, there's a simple parameter to pass to the zpool utility that would have saved the OP from dropping his pool:

    zpool clear tank

ZFS does actually come with a decent set of tools for fixing and clearing faults - which I'm starting to realise isn't as widely known (possibly because people rarely run into a situation where they need it?)

Anecdotally speaking; I've ran the same pool for years on consumer hardware and that pool has even survived a failing SATA controller that was randomly dropping disks during heavy IO and causing kernel panics. So I've lost count of the number times ZFS has saved my storage pool from complete data loss where other file systems would have just given up.

[ Parent | on HN ]

[deleted]

Re:71 TiB DIY NAS Based on ZFS on Linux by laumars on Monday September 01, 2014 @11:53AM (#8253424)

Indeed, however that's all dropping and recreating the pool would do as well. The file would still be corrupt.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by kogir on Sunday August 31, 2014 @07:55PM (#8250923)

I agree completely. Much better to know than not. But having no option to knowingly just lose inconsistent data and move on is a bummer.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by dsturnbull2049 on Tuesday September 02, 2014 @09:51AM (#8257500)

A corrupt file in ZFS indicates multiple broken disks. Copy the file in from a backup, use raidz3 and/or set copies=2 for critical zfs volumes. https://blogs.oracle.com/relling/entry/zfs_copies_and_data_p...

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by insaneirish on Sunday August 31, 2014 @07:15PM (#8250776)

Do you have ECC memory on that host? That corruption is not normal and suggests a problem likely with your hardware.

Do you have snapshots? Can you read that file in a snapshot?

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by Filligree on Sunday August 31, 2014 @07:27PM (#8250835)

I've had the same thing happen on a similar system, with ECC. In that case I eventually tracked it down to a dodgy power supply, but the damage was done... it was time to dump and reload.

Corruption happens. A good filesystem needs to cope.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by kogir on Sunday August 31, 2014 @07:58PM (#8250928)

The memory is ECC. No data I care about was affected. I'm just surprised there's no "do whatever you need to make it consistent so I can move on with life" option and instead I have to bring up a whole new pool.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by insaneirish on Sunday August 31, 2014 @10:20PM (#8251258)

You should post on the illumos zfs-discuss mailing list. The most amount of experts will be there to provide comments. I just saw the hostname of the machine; your "clout" may get you some attention. :-)

You very well may have some flaky hardware which is causing the issue, in which case, it probably needs to be fixed sooner than later. And maybe with some magic, you can get the pool into a happy state without rebuilding.

[ Parent | on HN ]

Re:71 TiB DIY NAS Based on ZFS on Linux by takeda on Monday September 01, 2014 @03:57AM (#8251906)

If you are having random errors like that it might be a sign of something bigger.

As mentioned in my other response I had random bit flips in memory. After I replaced my broken NIC crashes stopped. But I got a nasty surprise when I removed a filesystem - kernel panic due to failed assertion, and then kernel panic for the same assertion every single time I tried to import the pool.

I suspect it possibly could be resolved somehow without losing data, but decided to simply restore it since my backup was relatively fresh.

I suspect you might have something similar to what I experienced, and even though you use ECC memory I doubt it would help much when corruption is happening on a bus.

[ Parent | on HN ]