Currently running stateful apps on a DC/OS cluster + HA DBs. Fairly straight forward to get stateful working now + there are libs to migrate data around to chase your apps/services when/if Mesos/Marathon relocates them (e.g. after a service restart/crash) if you need your data co-located with a managed app/service.

The thing I'm most interested in exploring with K8+DC/OS is having DC/OS manage a couple of K8 instances so we can isolate 'virtual' clusters for various envs/apps. I suppose you could do this already with Marathon (the DC/OS built-in container manager), but we're not. Psyched to benefit from the K8 community + have the underlying DC/OS VM control/management plane.

Thanks, can you elaborate on the libs available to "migrate your data around"? Does Mesosphere reschedule your DB to another node that has an equivalent persistent and reserved storage(SSD etc.) volume configured on it?

Are you using Portworx for this?

> "can you elaborate on the libs available to 'migrate your data around'? ... Are you using Portworx for this?"

The tool I had in mind was 'REX-Ray'[https://mesosphere.github.io/marathon/docs/external-volumes....].

That said, we're not actually doing the 'chasing db' config. Instead we run a HA Neo4j DB deployment as a Marathon service pegged to a handful of nodes each with local persistent volumes allocated to Neo4j. I.e. we can allocate a % of a node's resources to 'static' Neo4j deploys, and then let Marathon dynamically manage any remaining free resources on the nodes. Our other services then use the Marathon service DNS to look up the Neo4j service for read/write.

Portworx looks cool too -- will need to investigate.

Also, the DC/OS documentation is quite good in general if you're looking to dig in on this: https://dcos.io/docs/1.9/storage/

> "Does Mesosphere reschedule your DB to another node that has an equivalent persistent and reserved storage(SSD etc.) volume configured on it?"

Yes/it can, but in that config you're booting up new/empty storage volumes. Obviously not what you want for many core persistence requirements though great for caches. We'll probably opt for this config near-term for our web-server SSR cache.

Michael from Portworx here. Thanks for the shout out. Just for some context, we just announced a partnership with Mesosphere today to help accelerate adoption of DCOS for stateful services [0] in fact. We handle the automation of all the state management mentioned above, not just volume provisioning. Our customers include big companies like GE and Dreamworks but also a lot of smaller companies. You can use PX-Dev[1] for free up to 3 nodes. Would love feedback.

[0] http://m.marketwired.com/press-release/mesosphere-partners-w...

[1] https://docs.portworx.com

Cool -- We're a small operation atm so will take a look at that dev tier. For more context, we also do some block storage off-cluster in GCE.

Thanks for the explanation. I've heard of Rex-RAY but I thought that was a vendor-specific solution(EMC.) Maybe that has changed?

>"We'll probably opt for this config near-term for our web-server SSR cache."

What is an SSR cache?

Yes, Rex-Ray is EMC specific. Take a look at Robin Systems (https://robinsystems.com/) for stateful containers. They have examples of running hadoop, cassandra, mongodb, etc all on commodity hardware.

That's not really accurate I think. RexRay is also supporting non-EMC solutions: https://github.com/codedellemc/rexray#storage-provider-suppo...

Mesosphere co-founder here.

This is correct: "DC/OS is better for running stateful services and then you can use K8 to run your stateless services"

Data services run directly on DC/OS via application-aware schedulers. They have the operational logic for how to bring up say a Cassandra cluster correctly, how to upgrade it to a new version without breaking it, change config, scale up, etc. All things you usually have to figure out yourself. When you run k8s on DC/OS you get these same benefits.

Thanks for the clarification.