News for hackers. Stuff that matters.
Re:Simple Systems Have Less Downtime by rsync on Tuesday March 03, 2020 @10:56AM (#22475537)
I gave a talk on this subject at CU last year and have, one way or another, spent my entire professional life thinking about this topic.

I agree wholeheartedly that simple systems have less downtime.

I would like to add a line from the talk that I give:

Simple systems fail in boring ways. Complex systems fail in fascinating, unexpected ways.

rsync.net storage arrays typically have multi-hundred day uptimes. But across our entire network, our actual aggregate uptime is unimpressive - perhaps 99.99% ? Our SLA dictates 99.95.

But when they do fail they fail in very, very boring ways. There are usually zero decisions to make in response to a failure.

Re:Simple Systems Have Less Downtime by carterehsmith on Tuesday March 03, 2020 @09:53PM (#22480808)
"simple systems have less downtime"

That is not much of an insight. It is as insightful as saying "water is kinda wet.". Well... sure it is.

What we need to deal with, is not "make a simplest system".

Rather, we need to deal with: "build a system that does A, B, C, ... and so on". Now, if you can do all of the above and make it simple... awesome. But if you cannot do all of the above, but the system is simple.... that is useless.

Re:Simple Systems Have Less Downtime by rsync on Wednesday March 04, 2020 @11:09AM (#22484910)
"Rather, we need to deal with: "build a system that does A, B, C, ... and so on". Now, if you can do all of the above and make it simple... awesome."

I agree with this. It is a good point.

Some systems and protocols have very difficult requirements to meet and different constituencies driving those requirements - it's not always possible to implement simple and elegant solutions.

Re:Simple Systems Have Less Downtime by badlogin78 on Wednesday March 04, 2020 @10:16AM (#22484408)
beware: for most is not that obvious. For most developers modern, extensible, layered comes before simple.
Re:Simple Systems Have Less Downtime by klysm on Tuesday March 03, 2020 @03:35PM (#22478470)
rsync.net is built on ZFS which is anything but simple. Having worked with multi-petabyte ZFS systems, you can run into some seriously hard to track down issues.
Re:Simple Systems Have Less Downtime by rsync on Tuesday March 03, 2020 @04:31PM (#22478901)
"rsync.net is built on ZFS which is anything but simple. Having worked with multi-petabyte ZFS systems, you can run into some seriously hard to track down issues."

Obviously I have a few things to say about this ...

First, zfs was considered production worthy on FreeBSD and put into fairly widespread use as early as ... 2008 ? 2009 ? We could have made very good use of it as we were running into all kinds of limits and corner cases with UFS2 and very large filesystems with hundreds of millions of inodes.

But we waited until late 2012 to do our first deployment and it took us about six years to finally deprecate the last UFS2 systems. We did this out of an abundance of caution and a desire to see things shake themselves out over several major releases of FreeBSD.

As for the complexity of ZFS, our previous architecture had 3ware RAID cards and all of their firmware and complexity sitting between the drives and the OS. In a way there was a beautiful elegance (in my opinion) in giving the OS a single drive:

    newfs /dev/da0
.... and that single drive just happened to be 40 TB in size and the OS has no idea what's going on underneath ... but there is a lot of complexity inside a full-blown RAID card and a lot of ways for drives to interact weirdly with it - especially when the drive is X years newer than the latest firmware for the card.

I find that on the very deepest, hardware level, handing over raw disks to ZFS and letting it manage them is simpler. We remove a fairly complex piece of hardware and accompanying firmware (since our HBAs are "degraded" to dumb "IT" mode).

Further, all of the "bolt-ons" of UFS2 that we absolutely relied on, such as quotas and snapshots, are elegantly built into the filesystem from the very lowest levels.

We ran rsync.net (and its predecessor) on UFS2 for 11-12 years and I can tell you from the perspective of both day to day management and middle of night firefighting, ZFS has made our lives much simpler.

Re:Simple Systems Have Less Downtime by throw0101a on Tuesday March 03, 2020 @03:44PM (#22478547)
Is ZFS more or less complicated than mdraid+LVM+ext4?
Re:Simple Systems Have Less Downtime by vthriller on Tuesday March 03, 2020 @04:10PM (#22478745)
I'd say it's more complicated even for simple ZFS pools without any redundancy or fancy features. My favorite illustration on this topic would be figure 2 from "Reliability Analysis of ZFS" (PDF): http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf

I still miss some of the features that are hard (if not impossible) to replicate in layered storage systems, like Merkle tree checksumming, but near-nightmare that low-level debugging of ZFS was actually turned me away from that filesystem.

Re:Simple Systems Have Less Downtime by sysashi on Tuesday March 03, 2020 @02:43PM (#22478042)
sorry for offtopic.

there a broken link that I was interested to read on. its on https://www.rsync.net/resources/howto/rsync.html ctrl+f "rsync snapshots are detailed here"

I think given the topic, someone is expected to figure it out on their own (remember linux cake?), but I feel a bit lazy after a work day.. sorry

Re:Simple Systems Have Less Downtime by rsync on Tuesday March 03, 2020 @04:36PM (#22478934)
Sorry - that is in purpose ...

The definitive page for rsync snapshots has been, and always will be, here:

http://www.mikerubel.org/computers/rsync_snapshots/

... we actually don't want people to do rsync snapshots anymore because ZFS snapshots are much more efficient and use up less of their rsync.net account.

If you change one bit of a file, the rsync snapshot method will cost the entire size of that file, since it's a hard link and is either broken or not broken.

But if you just do a "dumb" sync to us and change one bit in the file, your ZFS snapshot will take up just one more bit.

So not only do you get a much simpler backup script - you can really just do a dumb sync to us and let us rotate your snapshots - but you get more efficient space usage for changed files.

Re:Simple Systems Have Less Downtime by sysashi on Wednesday March 04, 2020 @02:09AM (#22481787)
Thanks for the link and explanations, appreciate it!

I've read that page (i linked) inattentively, re-read 3 times that bold text next to the broken link and still asked that question... :facepalm: