What I see is a lot of complex network filesystems, vendor-specific solutions and gateway protocols to expensive SAN solutions, which are already chalk and cheese in terms of features and performance.

Arguably one of the best features of unix-style systems is support for arbitrary mount points, filesystem drivers and (network or local) blockstores. Storage is, essentially, a well-solved problem at the OS level. The fact that this option is marked "single node testing only – local storage is not supported in any way and WILL NOT WORK in a multi-node cluster" raises eyebrows.

By choosing to expose individual remote storage model semantics as Kube-level PV drivers instead of just leaving this to the OS, what I would argue we essentially see here is the legacy of a cluster orchestration system that came out of Google... a system optimized for large, homogenous, dynamic workloads to provide organization-internal IaaS, and not reduced feature-set systems with simpler architectural properties (eg. no multi-client network-aware filesystem locking).

I would argue that, in fact, what many people actually want is simpler, and the current pressure to use 'one size fits all' cluster orchestration systems with a high minimum bar of functionality and nodecount (read: minimum hardware investment) is misplaced. At the very least, there's some legitimacy to this line of thinking.

Since suggesting your company is not in the same class of the companies that see literally billions of unique users every day, and thus may not need such overcomplicated solutions, is sure to make your boss irate, it's a good idea to familiarize with whatever new hotness has Facebook or Google's name attached to it.

Your clueless colleagues will race each other to announce the latest Google/FB engineering blog post in Slack so they can look the smartest and then convince your boss that since your Google-dom will be upon you tomorrow, you must adopt HotNewStuff today. This impulse is behind the proliferation of Hadoop and "Big Data", containers and orchestration, and MongoDB and NoSQL. All of these are useful tools that are valuable and good as necessary, but widely abused because people who don't really know what they're doing think this will give them an out.

You'll be stuck maintaining something interesting but really not mature or production-ready like k8s for years, just about long enough for it to become smooth and stable, at which time something else will come along to repeat the cycle. :)

Personally, while there is undoubtedly a convenience factor with being pure EC2 and a cool factor with k8s, I think 80% of our stuff would be better off in the racks (which included a couple of hypervisors, so we still had some cloud-style flexibility and could do things like auto-scaling).

Obviously I imagine you know other tools better and it depends how you do it, but kube gives you a lot more by default. Arguably more importantly, I can lift and shift kubernetes and put it in any cloud or on-premise. I'm not really sure what the benefit of running a VMs would give you, other than possibly live migrations.

Disclosure: I work at Google on Kubernetes

First issue for me: the recommended way to run k8s for local testing, etc., is minikube. I run a hybrid Windows-Linux desktop env since June (full-time Linux for 10+ years before that), where Windows is the host OS and my Linux install is running as a VBox guest with raw disk passthrough. I have it configured essentially so that Windows acts like a Linux DE that can run Photoshop and play games, while I do all my real work through an SSH session to the local VM, which is my Linux install (and which I can boot into natively if desired, but dual-booting always impairs workflow, which was the reason I switched to this setup in the first place; previously, I would reboot into Windows maybe once a year even though there were games and things I wanted to try and photo editing in VMs hosted on my Linux box was painfully slow).

This means that minikube, itself dependent on VMs to spin up fake cluster members, won't work because VM hardware extensions aren't emulated through VirtualBox's fake CPU. So that's the first hurdle that has stopped me from tinkering more seriously with k8s clusters. I know there is "k8s the hard way" and stuff like that too, but it'd be really nice if we had a semi-easy way to get a test/local k8s up and running without requiring VM extensions, as I imagine (but don't actually know) most cloud rentals don't support nested VMs either.

Besides this big hurdle to starting out, many of the issues are high-level complexity things that create a barrier to entry more than things that actively get in the way of daily use once you understand them.

For example, we have 3 YAML files per service that need to be edited correctly before something can be deployed: [service]-configmap.yaml, [service]-deployment.yaml, and [service]-service.yaml. We have dozens of services deployed on this cluster, so we have hundreds of these things floating around. They're well-organized, but this alone is a headache. The specific keys have to be looked up, they have to be in the right type of configuration; if something that is supposed to be in the configmap is in the deployment file, k8s will be unhappy, the right env variable won't get set (more dangerous than it sounds sometimes), the wanted shared resource won't get mounted correctly (and my experience is that it's not always obvious when this is the case, and the mount behavior is not always consistent), or whatever. Keys must be valid DNS names, or something like that, because etcd, which runs under the covers here somewhere, doesn't accept names that would be invalid DNS entries. This means no underscores. There's nothing wrong with any of that per se, but it's a lot to wield/remember.

I also remember mostly thinking the errors related to k8s configurations and commands were unhelpful. For example, it took me a long time (a frustrating 60-90 minutes probably) to realize that `kubectl create --from-file` wasn't reading in my maps as config structures, but rather as literal strings. This seems like something that should've been made obvious through something like a warning on import ("--from-file imports your map as a literal; if you want to parse the contents and use it as a config, please use `apply -f`" (and `apply -f` means apply the config read and parsed from the file, not "apply with force", while `create --from-file` means "create a literal string as a resource instead of parsing and creating this config as a config object; however, be careful with kubectl apply, because it will try to silently merge existing configs with new values, which is sometimes helpful, and sometimes can drive you nuts if you forget about this behavior; I dunno if kubectl delete configmap my-configmap.yaml and recreate is always feasible or if that would give dependency conflicts or what?)).

To deploy, kubectl apply -f changed-yaml.yaml, which sometimes does and sometimes doesn't clean up the running pod (service configuration thing? or is it a matter of which config type I'm applying, cm, deployment, or service?), `kubectl delete pod old_pod_id` if not automatically reaped, restarting is automatic under our config after a delete which I'd guess is configurable too, then you have to `kubectl list pods | grep service_name` to get the new pod id, and `kubectl logs pod_id` to get the logs and make sure everything started up normally, though this just shows the logs output by the container's stdout, not necessarily the relevant/necessary logs. Container-level issues won't show on `kubectl logs`, but require `kubectl describe pod pod_id -o wide`.

Then you have to `kubectl exec -it pod_id /bin/whatever` to get into the right container if you need to poke around in the shell (and I know, you're not supposed to need to do this often). Side note here, tons of people trying to containerize their apps that run on Ubuntu or Debian today onto Alpine, another mostly-unnecessary distraction, and seems to result in just grabbing a random container image from Docker Registry that claims to provide a good Ruby runtime on Alpine or something without looking into the Dockerfile to confirm, which IMO is a much larger security risk than just running a full Ubuntu container.

Lots of extended options like `kubectl get pods -o [something]` are non-intuitive. I guess they're JSON pathspecs or something like that? Again, that probably makes sense, but it's pretty unwieldy. Often have to do `kubectl describe pod pod_id -o wide` to get useful container state detail.

When a running pod was going bananas, we had to `kubectl describe nodes`, again a long and unwieldy output format, and we have to try to decipher from the 4 numbers given there what kind of performance profile a pod is encountering. This leads us into setting resource quotas to make sure that pods on the same node don't starve each other out, which is something I know the main k8s guy has had to tinker with a lot to get reasonably workable.

Yes, we have frontend visualizers like Datadog that help smooth some of this over by giving a near-real-time graph with performance info, but there's still a lot of requisite kubectl-fu before we can get anything done. I also know that there are a ton of k8s and container ecosystem startups that claim to offer a sane GUI into all of this, but I haven't tried many yet, probably because I'm not really convinced any of this is necessary as opposed to just cool, which it undoubtedly is, but that's not how engineers are supposed to run production environments.

I mean all of this doesn't even scratch the surface, and I know they're not huge complaints, but they just speak to the complexity of this, and a reasonable person has to have some incentive to do it besides "It makes us more like Google". Haven't talked about configuring logging (which requires cooperation from the container to dump to the right place), inability to set a reliable and specific hostname for a container in a pod that will persist through deployments, YAML/JSON/etcd naming and syntax peculiarities in the deployment configs, getting load balancing right, crash recovery, pod deployments breaking bill-by-agent services like NewRelic and Datadog and making account execs mad, misguided people desparetely trying to stuff things like databases into this system that automatically throws away all changes to a container whenever it gets poked, because everything MUST be using k8s, since you already promised the boss you were Google Jr. and he will accept nothing less, and a whole bunch of other stuff.

All of this ON TOP OF the immaturity and complexity of Docker, which itself is no small beast, on top of EC2.

That's QUITE the scaffolding to get your moderate-traffic system running when, to be honest, straightforward provisioning with more conventional tooling like Ansible would be more than sufficient -- it would be downright sane!

SOOOOOOOOO ok. Again, I'm not saying there's anything wrong with how any of this is done per se, and I'm sure some organizations really do need to deal with all of this and build custom interfaces and glue code and visualizers to make it grokable and workable, and of course Google is among them as this is the third-generation orchestration system in use there. None of this should be taken as disrespectful to any of the engineers who've built this amazing contraption, because it truly is impressive. It's just not necessary for the types of deployments we're seeing everyone doing, which has nothing to do with the k8s team itself.

I'm sure that given the popularity of k8s, people will develop the porcelain on top of the plumbing and make it pretty reasonable here in the not-so-distant future (3-5 years). However, like I said in my original post in this thread, I don't think this is benefiting many of the medium-sized companies that are using it. I think, to be completely frank, most deployments are engineers over-engineering for fun and resume points. And there's nothing wrong with that if their companies want to support it, I guess. But there's no way it's necessary for non-billion-user companies unless you REALLY want to try hard to make it that way.

I could write something extremely similar to this about "Big Data". Instead of concluding with suggesting Ansible, we could conclude with suggesting just using a real SQL server instead of Hadooping it up with all of those moving parts and quirky Apache-Something-New-From-Last-Week gadgets and then installing Hive or something so you can pretend it's still a SQL database.

Is there a way to make over-engineering unsexy? That's the real problem technologists who value their sanity should be focusing on.

Also, if you aren't already a member, come join us in #sig-aws on the kubernetes slack - we're a group of Kubernetes on AWS users, mostly happy - and working together to figure out the pieces where things could be better!

Disclaimer: DevOps going through the same process (everyone wanting to move to Kubernetes/k8 because its the new hotness/"orchestrated containers").

Just to clarify this a bit: Persistent volumes as an API _resource_ in Kubernetes are independent of which node a container requesting them is scheduled on, which is why it makes little sense to have a host-independent host volume.

If you have your storage sorted out on the hosts you can use a "simple" volume to mount it correctly [1]. Scheduling can also be restricted to the correct nodes with that storage by using node selectors / labels.

1: http://kubernetes.io/docs/user-guide/volumes/#hostpath

Disclosure: I work at Google on Kubernetes

That's what I'm hoping for in 2017.

The model I want to see is fast network-centric access to replicated local data. Some vendors are pushing into this space now. 2017 will be exciting.

It HAPPENS to work in a single node cluster because the scheduler doesn't have a whole lot of wiggle room to do the wrong thing :)

I really hope that storage for K8S happens this year in a form that is simpler than Gluster/Ceph/etc and preferably integrated. Right now we're using NFS and it's ok for the simple applications but I right now I wouldn't dare deploy my databases in K8S and feel good about it.

What gives me hope is the speed the K8S is moving forward and the superb experience we had thus far.

Since these applications are fundamentally not designed for something like kubernetes what everyone seems to be asking for is Kubernetes to provide network storage that has the good things from NFS but faster. I could be way off base here but asking for a simple and performant solution to this problem from projects like kubernetes just doesn't seem reasonable. Network storage, particularly the type that provides traditional filesystem semantics, is a hard space all on its own.

To put it another way, it sounds like people are asking kubernetes to take single node applications dependent on traditional filesystems and provide a magical performant network storage layer to make those failover seamlessly between kubernetes nodes with no (or tunable?) data loss. For those asking for this how would _you_ go about creating such a system? Just thinking through that a little should make it clear that what you're asking for isn't just a solution to a very difficult problem, it's a solution that is, I think, worth quite a lot of money to the person/group that solves it elegantly.

I would genuinely love to be proven wrong here so by all means destroy my argument with extreme prejudice.

It seems to provide exactly the features required (fast local access with failover), at the expense of a window of lost data (since the replication isn't synchronous and confirmed).

[1] http://www.bolthole.com/solaris/zrep/

Disclosure: I work at Google on Kubernetes

Why would somebody downvote asking a question?

If your systems support software level replication (Elasticsearch, Cassandra, MySQL, MongoDB all do) then why do you need persistent storage? You just need container scheduling anti-affinity and enough replicas.

You only need persistent storage for systems which don't support that replication. Ceph can certainly be deployed as performant for DB workloads.

You say "Cinder has the stench of OpenStack" but Cinder is just a Python based webapp which povides an API to arbitrary storage backends (Ceph RBD, iSCSI, NetApp ONTAP, whatever). How can it be "better now"? It doesn't provide storage on its own. If your ops team was using the default "proof of concept" LVM backend then I could see how you might get a bad impression but that just means your ops team doesn't know much about OpenStack.

Am I missing something obvious?

First off, the replication thing. It is true that ES, C*, and Mongo replicate within their cluster mostly automatically. However, this is not without cost. It takes non-trivial amounts of network capacity, disk I/O, and CPU cycles to migrate shards from a failed (or downed) node to a newly stood-up node. Often, many GBs must be moved and for something like ES, where shard replicas reside on many different nodes, that means much of your cluster feels the impact of this. The cluster can heal, but healing isn't easy.

Why would a cluster node go down? It's not always hardware failure. CoreOS regularly self-updates and reboots itself without intervention. In a Kubernetes cluster, this is a non-event because pods are simply rescheduled elsewhere the the degradation is momentary. If we were talking about 300 GB of persistent data, though, that's a serious amount of data that will get reshuffled every time there is a node reboot, especially when you consider that an Elasticsearch cluster may span dozens of physical nodes and experience dozens of node reboots in the course of a normal day. Maybe we could hack something that would disable shard reallocation in ES (there's a setting for this) when scheduled reboots happen but that's pretty hacky. Besides, ES is just one of a number of different datastores in use at my workplace.

As for Cinder, it's reliant on OpenStack APIs which (at least as of Juno) are reliant on things like RabbitMQ. We've seen a number of OpenStack failures due to RabbitMQ partitioning and split-brained scenarios. We're also back to the disk-on-network problem again: SCSI backplane ---ethernet---> client will never be as fast as local disk.

I'm talking about replication, not sharding though. If the data is actually lost then you have to bear the penalty of re-replicating it to match your replica count regardless, there's no magic wand here to do with "persistent storage". If the data isn't actually lost (e.g. due to CoreOS automagic reboots) then you absolutely should be putting the cluster into maintenance mode until the reboots are complete.

> As for Cinder, it's reliant on OpenStack APIs which (at least as of Juno) are reliant on things like RabbitMQ. We've seen a number of OpenStack failures due to RabbitMQ partitioning and split-brained scenarios.

Still pretty confused when you mention OpenStack. Cinder doesn't rely on OpenStack APIs per se, it provides an OpenStack API (for block storage). RabbitMQ clustering has longstanding issues with partitions which are mentioned explicitly in the documentation, nothing to do with OpenStack, everything to do with Erlang MNESIA DB. Any decent OpenStack team has learned by now to use singleton RabbitMQs with a master/slave configuration loadbalancer (i.e. haproxy) in front.

> We're also back to the disk-on-network problem again: SCSI backplane ---ethernet---> client will never be as fast as local disk.

Right. But wasn't the comment about persistent storage? You're never going to have persistent storage in your k8s cluster that magically avoids that problem, so not really sure what the point is here.

http://cloudscaling.com/blog/cloud-computing/killing-the-sto...

Really?

https://www.spec.org/sfs2008/results/res2011q4/sfs2008-20111...

And those are nothing compared to modern systems.

We need something built on common PC chassis, either as distributed local storage or some type of high speed interconnect.

Banks and hospitals don't.

Is there any reason why you are trying this at all?

The biggest issue people have with NFS is when they roll their own. To get performance they allow asynchronous writes | but this means the write cache can potentially be lost during a power failure. Enterprise NAS systems like the NetApp referenced above have battery backed cache so writes are never acknowledged until they're in a secure medium.

https://github.com/NetApp/netappdvp

The one thing I have found on Mesos that I really liked are persisted volume resources:

http://mesos.apache.org/documentation/latest/persistent-volu...

I was hoping K8 has something similar but it didn't the last time that I looked.

Edit: I just looked at your 1st link. I see the PV docs, none of those fit my use case(local) unfortunately.

I'll just state the obvious that you're very much correct about stateful storage still being the most immature aspect of the Kubernetes and cloud native story, but there are definitely tons of folks working on it. And in the meantime, people are succeeding in production using last-generation solutions on bare metal or provider offerings in the cloud.

I'll make a quick pitch that if you are a cloud native end user interested in engaging with the community, please email me about joining CNCF's end user board (my email is in my profile). We just dropped the price from $50 K to $4.5 K, and it now includes 5 tickets to CloudNativeCon/KubeCon ($1.7 K and 2 tickets for startups).

[0] https://github.com/cncf/landscape

Many new databases (especially NoSQL ones) already support clustering and rely on being tightly coupled to specific nodes/machines in order to work. I think K8s currently makes this a bit too difficult - They need to improve support/documentation for using host storage for these kinds of clustered DBs.

NFS doesn't make sense for storing structured data because it doesn't know what the best way to partition/search your data is going to be (the directory tree structure isn't always what we need) - I think that this can only be solved at the DB layer unfortunately.

The entire existence of stateful containers depends on having network storage at hands.

The only good network storages are GCE and AWS volumes, which are proprietary trade-secret technologies only available there.

If you want to play it old-school. You can run virtual machines with VmWare on bare metal servers with SAN disks (iSCSI/Fibber Channel). The Vm can be hot migrated from an host to another. It works and it's been battle tested for almost a decade. (IMO: Docker is not only new but a toy in comparison to that).

Ganeti has been battle-tested for about a decade too, supports an assortment of storage backends, including some clustered ones like Ceph, can do live migration between hypervisor nodes, and it's a nicely maintained Python app with some Haskell parts in it.

ganeti came out of my team at google in the mid 2000s, fwiw. I did not work on it, but I've certainly used it. It's pretty nice.

Actually Kubernetes is starting to work on support for persistent local volumes; we know the lack of this feature is a significant barrier for running some stateful applications on Kubernetes, particularly on bare metal. The concrete proposal for how we are thinking to do it is at https://github.com/kubernetes/kubernetes/pull/30044

The high-level feature requests are at https://github.com/kubernetes/features/issues/121 and https://github.com/kubernetes/kubernetes/issues/7562

(Disclosure: I work on Kubernetes at Google.)

IMO: It's a marketing and usage problems. You should re-focus people on running exclusively stateless containers. Sell the strengths of containers, what it's good at and what it's meant to do. Containers = stateless.

Stateful containers are an hyped aberration. People barely get stateless containers working but they want to do stateful.

Go see the outside world => They have none of your internal tech and services. Stateful containers do not exist there. "Containers" means "docker" which is experimental at best.

[1] http://www.clearskydata.com/

I don't mean to be negative but I'm having some trouble seeing how the ClearSky feature set justifies assuming what looks like an existential risk to business-critical data on the service. Interested in your thoughts on this.

edit: typo

Feel free to prove me wrong but I don't think that this is a reasonable solution for backing storage for a database.

With ClearSky, even for workloads that don't fit in the edge cache (which we believe are few), you'll still see single-digit millisecond random read (and write) latencies. This is made possible by our points of presence (PoPs) located in each metro area we serve. These PoPs house the lion's share of data in a private cloud, and are connected to each customer site with private lines with sub-millisecond latency.

In other words, the speed of light is very fast when it goes in a straight line with nothing in its way ;) While we do have some secret sauce in the data pipeline, it is because we own the network to the customer that we can provide the performance we do.

(Fancy PDF with a few more details here: http://cdn2.hubspot.net/hubfs/445689/2015_assets/ClearSky-Da...)

Several members of our team were core developers of EqualLogic (pre-Dell buyout). We have significant investment from Akamai. I promise you this is not snake oil.

Aside, I can't express how validating it is how much you (and others, given the downvotes) disbelieve me. It makes me quite proud to have helped develop a service considered so impossible that it is written off as black magic. It does make it hard to market the damn thing though ;)

[1] http://www.clearskydata.com/clearsky-global-storage-network-...

First, this is a technical audience. Plenty (most?) of us couldn't give a shit about what marketing says. I believe about half of what comes out of the mouths of sales/marketing folks -- for good reason. We don't care what Gardner or some firm you paid to write up a report says.

Personally, I have a real dislike of sales/marketing folks and I will avoid them at all costs... so, no, I don't want to give you my name or e-mail address. I don't want your people calling me, interrupting real work. I don't want to view your webinar. I want to look at the technical details -- the facts -- and decide for myself and then, quite possibly, completely forget I ever heard about your company and go about my day.

Last, don't fool yourself. The downvotes aren't "validation" -- at all -- but, hey, go on living in your fantasy world. If it really was as awesome as you seem to think it is, it wouldn't be hard to market. To the contrary, the damn thing would sell itself and you wouldn't even need a marketing department.

http://docs.aws.amazon.com/storagegateway/latest/userguide/G...

* ClearSky has a dedicated private network to every customer ensuring low latency.

* ClearSky fully manages disaster recovery; you lose no data if your datacenter is destroyed.

* ClearSky provides transparent instantaneous data mobility: you can move volumes to any other data center in a matter of seconds.

* ClearSky provides and manages the edge cache.

All of the above must be provided by the customer with Amazon's offering.

That said, we hear you, we're contemplating options for local-disk volumes.

https://www.supermicro.com/solutions/storage_ceph.cfm

We went for upgraded network capacity though, 20 Gbit/sec cluster backend, 20 Gbit/sec cluster frontend...

12 * 8 TB drives, with 800 GB NVMe for the Ceph journal. Fast, large Ceph journal was key.

Total installation was about 3 PB raw, that is 1 PB useable with replication size 3. 33 Ceph OSD nodes, 3 Ceph monitor nodes and Juniper low latency switching using the QFX5100.

Full IPv6 network on both frontend/backend. 11 nodes per rack, each rack being it's own /64 routed domain. 3 racks.

I'm no longer doing contract work for the company, but last I heard they were expanding it out to 6 racks with an additional 3 PB raw capacity added on because of growing datasets.

It's an OpenStack cluster that is connected to this Ceph cluster, 40 Gbit/sec storage backend network, with 40 Gbit/sec front-end that VM's have all their traffic on. So storage and standard traffic don't mix.

The performance and IOPS even virtualized were enough that the entire company is moving their bare metal databases to VM's. I am unable to disclose IOPS or Oracle database performance due to contractual obligations unfortunately.

I'd be curious to understand your storage requirements as a production Kubernetes user. What would you like to see for performance, cost and RPO/RTO?

https://en.wikipedia.org/wiki/Data_virtualization

https://en.wikipedia.org/wiki/Software-defined_storage

do you use hostpath right now ?

It's my #1 goal for 2017: figure out persistent volumes for Kube.

These are for cloud foundry nominally but (yay actually collaborating foundations) the open service broker API ( https://www.openservicebrokerapi.org/ ) allows you to hook these into Kube and (the best part) standardize how your CI/CD pipelines manage the lifecycle of these services independently of whether they're backed by Openstack today or Kubernetes tomorrow.

Persistent volumes for these sorts of services on Kube will require the new PetSet primitive to mature.

[0]: https://objectivefs.com