News for hackers. Stuff that matters.
Re:XtreemFS – Fault Tolerant Distributed File System by fh973 on Thursday February 25, 2016 @02:17PM (#11176972)
One of its creators here. If XtreemFS looks interesting also check out Quobyte (www.quobyte.com), our next software storage system.
Re:XtreemFS – Fault Tolerant Distributed File System by DanielDent on Thursday February 25, 2016 @04:35PM (#11177999)
Quobyte does not appear to be a mature product.

Clicking 'get started' merely leads to a 'please let me waste hours talking with your enterprise sales reps' form.

A mature product should be possible for me to evaluate and try on my own, and I would rather not waste time doing so if I don't even know what it costs.

Re:XtreemFS – Fault Tolerant Distributed File System by fh973 on Thursday February 25, 2016 @04:51PM (#11178111)
You are totally right with your expectations. But in a startup you can't do everything at once and thus no direct download yet. If you contact us you'll talk directly to the developers and get a download link.

That said, Quobyte is in production for business critical workloads.

Re:XtreemFS – Fault Tolerant Distributed File System by robhu on Thursday February 25, 2016 @04:57PM (#11178161)
Of course you're right, a startup can't do everything -- so why not just put a download link up rather than require people to talk to you to get a download link? Given as a startup you're busy and have a lot to do, why not just post a link on your website?
Re:XtreemFS – Fault Tolerant Distributed File System by fh973 on Friday February 26, 2016 @04:08AM (#11180520)
Doing this thoroughly would mean that we would need to provide some form of support aswell.
Re:XtreemFS – Fault Tolerant Distributed File System by ymse on Thursday February 25, 2016 @05:05PM (#11178216)
How does Quobyte compare to Ceph?

Edit: From the product page the main advantage seems to be end-to-end checksums and an adaptive placement engine.

Architecture and performance comparisons would be very interesting, though I understand if you can't disclose that yet.

Re:XtreemFS – Fault Tolerant Distributed File System by craigyk on Thursday February 25, 2016 @04:36PM (#11178014)
How are the two related exactly? Is XtreemFS part of the core of the Quobyte product, or is it a dead-end, with all new development happening in Quobyte?
Re:XtreemFS – Fault Tolerant Distributed File System by fh973 on Thursday February 25, 2016 @04:39PM (#11178042)
XtreemFS is being maintained by the folks at Zuse Institute.

Quobyte is a new implementation and only shares the architecture blueprint with XtreemFS.

Re:XtreemFS – Fault Tolerant Distributed File System by seeing on Thursday February 25, 2016 @03:41PM (#11177623)
How are XtreemFS and Quobyte different than GlusterFS?
Re:XtreemFS – Fault Tolerant Distributed File System by fh973 on Thursday February 25, 2016 @03:59PM (#11177740)
Looking at the architecture, there are two major differences:

* XtreemFS has POSIX file system semantics and split-brain safe quorum replication for file data. With Quobyte, we pushed that further and have now full fault tolerance for all parts of the system, working at high performance for both file and block workloads (Quobyte also does erasure coding). GlusterFS replication is not split brain safe, and there are many failure modes that can corrupt your data.

* XtreemFS and Quobyte have metadata servers. This allows them place data for each file individually as it can store the location of the data. With Quobyte we pushed this quite far and have a policy engine that allows you configuring placement. When the policy is changed, the system can move file data transparently in the background. This way you can configure isolation, partitioning and tiering. GlusterFS has a pretty static assignment of file data to devices.

Re:XtreemFS – Fault Tolerant Distributed File System by notacoward on Friday February 26, 2016 @12:21AM (#11179932)
GlusterFS developer here. Sure, we have bugs just like anybody else. So do you. Nonetheless, just saying that GlusterFS replication "is not split-brain safe" is FUD. Neither is yours, not 100%. Likewise, "static assignment" is simply not true. We move files around to rebalance capacity, we have tiering, and users can explicitly place a file on a particular node (though the mechanisms for that are clunky and poorly documented). I've been fair and even complimentary toward XtreemFS in many blog posts and public appearances. I'd appreciate the same honesty and courtesy in return.
Re:XtreemFS – Fault Tolerant Distributed File System by fh973 on Friday February 26, 2016 @01:49AM (#11180152)
I am sorry that this came across this way. I did not intend to say bad things about GlusterFS but carve out where the technical differences are in the big picture.

Also I actually tried not to make any valuing statements. I am using the term split brain safety as technical term, ie. the P in CAP. My understanding is that GlusterFS does not have this in its system model and you and the documentation seem to support this: "This prevents most cases of "split brain" which result from conflicting writes to different bricks."

Quobyte generally (and XtreemFS only for files) does quorum replication based on Paxos, where split brain is part of the system model. They are CP and hence data is not always available but is always consistent for reads if the quorum is there. Like Ceph.

I am sorry that I missed the progress on placement. It seems like I need to catch up on what happened after the volume types.

Re:XtreemFS – Fault Tolerant Distributed File System by notacoward on Friday February 26, 2016 @03:02AM (#11180327)
"Split brain safety" is not a commonly used term, and even if that weren't the case I'd say it's not a term that should be thrown around lightly. Also, using Paxos or Raft doesn't guarantee split-brain safety, as aphyr has proven over and over again with Jepsen. So what we have is two systems that take different approaches to quorum and split brain and all that. It seems a bit disingenuous to throw stones at the older open-source project while ignoring the potential for the exact same problems in the newer proprietary one.

FWIW, I do think the current Gluster approach to replication is not sufficiently resistant to split-brain in the all-important edge cases. That's why I've been working on a new approach, much more like Ceph and many other systems - though few of them use Paxos in the I/O path. That's wasteful. Other methods such as chain or splay replication are sufficient, with better performance, so they're more common.

Re:XtreemFS – Fault Tolerant Distributed File System by troymc on Thursday February 25, 2016 @04:25PM (#11177929)
I gather that XtreemFS and Quobyte can tolerate benign but not byzantine faults. Is there such a thing as a byzantine fault tolerant file system?
Re:XtreemFS – Fault Tolerant Distributed File System by nickpsecurity on Thursday February 25, 2016 @06:10PM (#11178595)
Since 1999:

http://pmg.csail.mit.edu/papers/osdi99.pdf

Google byzantine fault tolerance filesystems to get quite a few more.

Re:XtreemFS – Fault Tolerant Distributed File System by troymc on Friday February 26, 2016 @12:51AM (#11179999)
Okay, but are there any in-production file systems implementing that (or any other BFT consensus protocol)?
Re:XtreemFS – Fault Tolerant Distributed File System by nickpsecurity on Friday February 26, 2016 @01:52AM (#11180161)
That I'm aware of, nobody wants to pay the price for that. Much like with high assurance systems in general. Demos, prototypes, and so on available with no uptake due to tradeoffs involved. Or maybe I.P. issues, too.
Re:XtreemFS – Fault Tolerant Distributed File System by fh973 on Thursday February 25, 2016 @04:37PM (#11178021)
Yes, no byzantine faults.
Re:XtreemFS – Fault Tolerant Distributed File System by nickpsecurity on Thursday February 25, 2016 @06:10PM (#11178599)
So, do you have any reasons for us to use XtreemFS vs its top competitors? What's the main benefits today that others don't have?
Re:XtreemFS – Fault Tolerant Distributed File System by protomyth on Thursday February 25, 2016 @04:10PM (#11177819)
XtreemFS is BSD, what is Quobyte?
Re:XtreemFS – Fault Tolerant Distributed File System by fh973 on Thursday February 25, 2016 @04:37PM (#11178027)
Quobyte is not open source.
Re:XtreemFS – Fault Tolerant Distributed File System by technimad on Friday February 26, 2016 @05:25AM (#11180711)
This looks a lot like ceph. Can you give a short overview of the differences?
Re:XtreemFS – Fault Tolerant Distributed File System by meapix on Thursday February 25, 2016 @02:00PM (#11176822)
> requires no special hardware or kernel modules

http://xtreemfs.org/quickstart.php

If not already loaded, load the FUSE kernel module:

> modprobe fuse

Re:XtreemFS – Fault Tolerant Distributed File System by pmalynin on Thursday February 25, 2016 @02:29PM (#11177076)
Except that FUSE is pretty mainstream.
Re:XtreemFS – Fault Tolerant Distributed File System by meapix on Friday February 26, 2016 @08:13AM (#11181163)
It was officially merged into the mainstream Linux kernel tree in kernel version 2.6.14. Anything before that is a special kernel module.
Re:XtreemFS – Fault Tolerant Distributed File System by RRRA on Thursday February 25, 2016 @06:08PM (#11178585)
How does it compare to Ceph and Tahoe-LAFS?

But more importantly, how are you using any of those? :)