A few major things we wanted to accomplish:

Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems.

Easy to work on code base. Building an OSS community around a complex technology is really hard. It shouldn't be underestimated that a code base that can be built on OS X, Linux, and Windows will have an advantage in getting people involved and maturing the project.

Designed for other storage application. Torus is backed by a distributed storage interface that is exposed via gRPC (with the opportunity for language bindings in lots of languages). Our hope is to let people build write-ahead log systems, object systems, or filesystems on top of this abstraction.

Flexibility built into the storage layer. Over time we want people to implement different hash maps or explicit configuration of storage layout. This is an abstraction that we have today and intend to build out over time with feedback from users.

Happy to answer more questions. This is a young project and we look forward to working with people and hearing about what they want out of a storage system. Overall, we want to build something with folks that is easy to work on and backs itself with technologies that are getting good operational understanding in the "Cloud Native" space.

I've managed to lose data with Etcd, and have regularly had issues with membership issues requiring maintenance. Meanwhile I've had Glusterfs volumes remain available for 5-6 years without maintenance at all.

To me at least, having it "backed by etcd" is a big red flag, not a feature.

I agree. I was also thinking of Ceph when I wrote my reply to Cantrill. Esp Ceph on XFS given maturity of both plus enormous effort that's gone into XFS. Sector/Sphere with UDT is also kick-ass. It took both of these a long time to get to where they were fighting all kinds of unforseen issues. They also tried to build on proven components that themselves were battletested for years.

So, this company is trading away stuff like that for custom components and Etcd? And for a component focusing on integrity and availability? Huh?

  I've managed to lose data with Etcd

How did you manage to lose data?

Use etcd in production and you will lose data. Bounce nodes, simulate power loss and you will lose data. Have you actually used it?

I don't remember the details, and to be clear this was not with the most recent version of Etcd at all - it was quite a while ago. I think and hope that whatever the problem was is no longer an issue. It has certainly gotten a lot better. My point is mainly that it is way too young to be something to trust important data to. We didn't either - what we lost was a cluster configuration that we could recreate from a combination of backups and redoing a handful of operations, but at the time it was scary to see, and prompted us to be very careful about what we put into Etcd going forward.

> Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems.

Being built atop something else doesn't mean it inherits all the stability of the underlying system; it's an upper bound not a lower bound.

It sounds like etcd is in the I/O path. Is that correct?

It's in the sync() path, but not in the I/O path.

Why is it in the sync path? That seems unnecessary, as well as bad for both performance and correctness (another thing in any path means another potential for partial failure).

This is not a great analogy since their first application is block storage, but think about a journaling file system. Typically file data is not journaled, but metadata is. By having a consistent view of the metadata, the entire filesystem (as far as you can interact with it) is consistent. That consistent journal is the same primitive that etcd is providing.

I don't think the two cases are analogous. First, block devices don't have any metadata to speak of. Second, filesystems have journals to "cover up" for the main-store operations being asynchronous and/or non-atomic. However, sync() is completely synchronous by definition - hence the name - so such "covering up" would be superfluous. There must be some metadata that's being written at etcd because it needs to be read from there later, but there's nothing in block-device semantics to require any such thing.

Thinking about it further, I think I can guess at what's going on here. The key observation is that there's no sync() at the block device level. It's a filesystem operation; block devices don't see it. Sure, there are queue flags and FUA and such, but those are different (and I'm not sure any of those exist in NBD). Where is this sync() path? I'm guessing it's internal on the data servers, to deal with data that's being buffered there. With both replication and erasure coding, correct recovery requires exact knowledge of what has been fully written where, and that's the kind of metadata I suspect is being put in etcd. There's not even necessarily anything wrong with it, unless updating that information only on sync() means that supposedly durable writes since the last (unpredictable to the client) sync could be lost on failure. I hope that's not the case.

Maybe I'll find time, in the midst of my work on an actual production-level distributed filesystem, to look at the code and see if my guess is correct.

Block devices (and NBD specifically) absolutely have a notion of sync(). We use sync() as the unit of write visibility. All writes up until a sync are effectively anonymous until a sync().

Please go look at the man page for sync(2), which is also the page for syncfs(2). It is explicitly a filesystem-level operation. Obviously, this will cause data to be flushed from the filesystem down to lower layers. Obviously, you can "sync" virtual (e.g. NBD or loopback) block devices by syncing the filesystems that contain their backing stores, but that's not the same thing. No filesystem, no sync(2). For block devices with no file-based backing store, sync(2) is inapplicable. Also, sync(2) is latency-inducing overkill if you're trying to ensure durability for anything less than all filesystems attached to a machine. More often, fsync(2) on the backing files is what you should be using.

> All writes up until a sync are effectively anonymous until a sync().

"Anonymous" means nothing in this context. Do you mean non-durable? First, please use the correct term. Second, if that is what you mean then you're probably doing it wrong. File writes are allowed to be asynchronous (unless O_SYNC and friends). Block device writes are expected to be synchronous, or at least to preserve order. This is exactly the kind of thing that needs to be thoroughly thought out before code is even written, and that thinking should be spelled out somewhere for people to help make sure all the nasty corner cases are covered. Your cart is way before your horse.

I kind of see this as a general trend where CoreOS is concerned they reimplement everything in house, even when open source solution might already exist - rkt, etcd, now this.

Rkt had good reasoning behind it, and I think it's reasonable to assume that a lot of the recent changes to Docker came because they forced Docker's hand.

For Etcd I'm of two minds. I like its simplicity. On the other hand, I have had far more problems with Etcd than I've ever had with e.g. Consul (though I dislike Consuls "and the kitchen sink" approach), up to and including data loss and more than once having to re-initialise clusters. I'm a couple of years of no further problems at least a way from trusting Etcd with my data.

For this? They have to have very compelling stories for why they couldn't e.g. modify Ceph or Gluster to achieve whatever it is that they want. And it's no clear what they want. Gluster for example is built as a stack of pluggable translators that is relatively easy to extend.

Even if they have a credible and compelling story for why we need something else, it will take years of production use by others before I'd trust any data to a new distributed file system.

Gluster and Ceph have those years. I've had a Gluster volume running without maintenance or data loss for more than five years. Even then it took several years before I truly started to trust it.

Have you ever thought the etcd issues was just because how you operate it? Or maybe you were running an old version of it? Have you reported to upstream? If there is a very common issue that you can meet frequently, it should have been fixed or you should have reported it.

I'm quite sure the etcd issues was "just because" of how we operated it. But the point is that I have had these issues with Etcd, using the default CoreOS configuration of Etcd, and I've not had these issues with Gluster despite having Gluster volumes running many times as long.

Gluster had many issues too when it was a young project, but they were fixed many years ago. I don't doubt that Etcd too will become solid enough for me to trust, but looking at part of the docs [1] we still have "gems" such as "Permanent Loss of Quorum Requires New Cluster" (you don't lose your data to that one, but it's painful nevertheless).

And yes, I've run into that one. Not "permanent" as in "we could never recover the machines in question", but "permanent" as in "we're not going to be able to get this sorted fast enough, so let's fail everything over". Everything else we run can handle that scenario easily. We know it means consistency issues, but if you run into that situation we have taken the decision that they are acceptable and will be resolved later if necessary. The last thing I need in a scenario like that is for some critical component to just refuse to run without requiring lots of manual intervention.

All we'd like would be to be able to say "ok, drop all the other members and pretend all is well, and yes we know what we're asking". There are a number of gotcha's like that with Etcd that sounds find until they bite you in the ass.

They'll resolve these things eventually, but storage is the very last place I'd be willing to take risks with it. So until they have years of track record of running flawlessly, with less hassle than Gluster or Ceph, and substantial advantages, I'll see it as the dangerous choice.

[1] https://coreos.com/etcd/docs/latest/runtime-reconf-design.ht...

Just to be clear: your complaint is etcd won't let you force the cluster into an inconsistent state to mask your poor infrastructure decisions and therefore etcd is a "dangerous choice"?

The "poor infrastructure decision" would be to assume that because there's a network split it is automatically going to be unsafe to, for example, keep scheduling containers in each data centre, even the minority, without knowing whether or not constraints have been put in place to ensure this is safe to do.

How many separate data centres do you want me to host in? Two is certainly insufficient, as if the data centre with the majority of Etcd nodes now falls of the network, you're SOL.

Three? Think you can't have two fall of face of the earth the first time? Been there. Redundant connections doesn't always help if there are major external connectivity issues and you don't have multi-million dollar budgets for connectivity alone.

Most small to mid size companies do not have the budget to run a system that is sufficiently redundant to be able to guarantee that they won't sooner or later be in a situation where either 1) the most viable partition is the minority, or 2) the Etcd cluster is centralised enough that too large parts of the cluster can loose access to it. When fighting large scale failures is the last time where you want to have to fight your tools to be able to convince them that you know what you're doing.

For those kind of situations, Etcd's lack of support for more fine-grained control of consistency or for intentionally breaking apart the cluster and letting them continue to operate independently when you know what you're doing basically makes it unsuitable for running multi-data centre clusters for anything important.

Which basically means any tools that assumes a single underlying Etcd cluster is appropriate as storage, for, say, a cluster scheduler (cough) becomes an inappropriate tool for those kind of setups.

Yeah, I've been smelling NIH for quite a while.

And it generally seems that, when they say "composable modular tools", one should read "includes dependencies on the rest of our stack". This isn't intended as harshness. It is just that every time a tool from COS seems interesting enough to look in to, it turns out that I wouldn't be investing in a tool, it would be investing in an ecosystem that duplicates a ton of what I'm already managing.

Kubernetes is a big exception to this, it seems. Before Kubernetes was announced, CoreOS was actively developing fleet. After Kubernetes, fleet development was largely halted and is now just a "distributed init system for low-level cluster components," e.g. a way to bootstrap Kubernetes itself. I asked the lead developer of fleet if they would have created it if Kubernetes had existed at the time, and he said no. Kubernetes has turned into a big part of the CoreOS pitch, and they've contributed a lot to the project. Giving up a project you started in house because you recognized that another organization's project fit the bill better is very far from NIH.

what is the existing alternative for storage in distributed container environments?

Ceph, OpenStack Swift, GlusterFS, OrangeFS, Lustre.

https://en.wikipedia.org/wiki/Clustered_file_system#Distribu...

https://en.wikipedia.org/wiki/Object_storage

If Lustre is the answer, you're probably asking the wrong question. Unless that question involves short life-span, massively parallel swap-outs from one weapons simulation to another.

I write this not as storage religion (of which there's far too much), but to warn away those who haven't experienced the many kinds of data (and stomach lining) loss that come with being a Lustre admin.

ObjectiveFS[1] is another option if you have access to an object store (e.g. S3 or GCS) from your containers.

[1] https://objectivefs.com

Add this to your list:

http://sector.sourceforge.net/

I swore I added it to the Wikipedia links. I'll have to do it again, I guess.

Have a look at Open vStorage (http://www.openvstorage.com). Some highlights: open-source, core is battle proven for more than 7 years, performance, scales across datacenters , unlimited snapshots, ...

(disclaimer: I'm an engineer there...)

Competition is good, even in open-source land. By the way, I'm still hoping for an open-source competitor for elasticsearch.

> I'm still hoping for an open-source competitor for elasticsearch

I thought Elasticsearch was open-sourced, and the GitHub repo says Apache v2. What am I missing here?

Nothing. It is open-source. But it needs an open-source competitor. There's too little happening at the open-source side of search.

Huh? It has a major competitor: Solr. I'm not sure what you mean unless you object because they both use lucene.

https://lucene.apache.org/solr/

What about Solr? They're both based on Lucene but I imagine it qualifies.

See also: Piston Cloud.