I wound up causing a major outage of my services and destroying my home dir

Re:I wound up causing a major outage of my services and dest... by Helmut10001 on Friday February 18, 2022 @01:30AM (#30382752)

While switching from Hardware Raid1 to ZFS recently, I had 1) sata cables failing on me and 2) a 8088-8087 PCI-Bracket corrupting half of the SATA connections through it. I would have never detected this without ZFS (scrub). All was fine for weeks. I did `fio` speedtests, `smart --long`, `dmesg` monitoring. Nothing. Until I ran the first scrub and 22 CRC read errors appeared in the 6x8TB RaidZ2. Much worse after switching drives and cables around. I spend like $400 for different cable-replacements, but finally identified the failing ones and all is good since then.

Now, I am more observant. Scrub once a month. Monitoring of SSD TBW, UDMA-CRC, Temperature, Watt, Fan-Speed in InfluxDB 2.0 with alerts set. Besides, ZFS mails all the important status info.

ZFS is really good for peace of mind, resilience and predictability, not necessarily for speed (but not exclusively, either), or your wallet.

[ Parent | on HN ]

Re:I wound up causing a major outage of my services and dest... by oarsinsync on Friday February 18, 2022 @10:24AM (#30386872)

Can you share what you use to capture the data (configs) and how you've configured your alerting?

[ Parent | on HN ]

Re:I wound up causing a major outage of my services and dest... by Helmut10001 on Saturday February 19, 2022 @01:15AM (#30394816)

I have written somee of it down here: https://du.nkel.dev/blog/2021-05-05_proxmox_influxdb/

Currently away, will update later.

[ Parent | on HN ]