News for hackers. Stuff that matters.
Re:Run Kubernetes on top of DC/OS by slap_shot on Wednesday September 06, 2017 @03:48PM (#15186580)
I recently reached out to Mesosphere to better understand the value and options available to my company if we switched to Mesos (and thus Mesosphere). We currently run Kubernetes across all major cloud providers and we'd like some help.

I was startled by how dogmatic they are about their pricing model: per node, annually. Our entire business model is running dynamically scaling data pipelines for companies. We spin nodes up and down programmatically based on a pipeline's history and a dozen other factors.

I had a lot of back and forth with them and eventually their CFO and they just could not understand the concept of a "node hour" or anything of the like. Eventually, it was determined if we paid the per-node annual price for our "average" (completely different story, but they calculated this about as wrong as you could have) usage and we went our separate ways.

The product seems really cool but as someone who runs a company that easily spin up thousands of nodes in the middle of the night for (for a grand total of $40/hr on GKE...) this pricing model just seems antiquated. I imagine that all companies will be driving their usage to this practice in due time. I really hope to see the industry change.

Re:Run Kubernetes on top of DC/OS by CoffeeDregs on Wednesday September 06, 2017 @04:19PM (#15186801)
Had an interesting pricing experience with them a couple of years ago. We were happy with our Mesos/Marathon cluster of ~100-200 machines but needed some acute support and we were in the same investment fund of DCVC. Got on the phone with ... Sales... No problem; you're a friend of ours because of DCVC; we'll get you special, friendly pricing on DC/OS...

Got an email a few hours late: $100,000 per year. Was fairly puzzled.

Two hours later, I got another email correcting the prior price: the special, friendly price was now $150,000 per year.

We migrated to AWS ECS and all was well.

Re:Run Kubernetes on top of DC/OS by pivotalpoint415 on Wednesday September 06, 2017 @06:07PM (#15187527)
A lot of software vendors charge $X,000 / server / year in subscription fees.

Redhat charges > $10,000 per physical server for their OpenShift and Pivotal gets for 100s of nodes often 7-figures. Remember just how much VMWare costs...

Re:Run Kubernetes on top of DC/OS by pnathan on Wednesday September 06, 2017 @04:25PM (#15186847)
I was examining a Mesosphere deployment a few years ago and also found the pricing model to be difficult in a cloud environment, but the salesperson I spoke with seemed more clueful than your description.

We chose to go otherwise for a variety of reasons.

Today, I think I would probably suggest to a Serious Company with a fully staffed ops/sre/devops team to run a pure Mesos play for the data systems with k8s on top or side by side for web systems.

Re:Run Kubernetes on top of DC/OS by rectang on Wednesday September 06, 2017 @04:39PM (#15186944)
> Mesos (and thus Mesosphere)

Mesos is an open source project. Would you consider any other vendors besides Mesosphere?

Re:Run Kubernetes on top of DC/OS by jmspring on Wednesday September 06, 2017 @04:44PM (#15186979)
Mesos and Marathon are open source, but there are features that the enterprise offering has (in addition to support) that the open source one does not.

Vendor wouldn't matter if someone needed those "enterprise" features.

Re:Run Kubernetes on top of DC/OS by don71 on Wednesday September 06, 2017 @04:45PM (#15186989)
DC/OS is also open source. What are your concerns with it?
Re:Run Kubernetes on top of DC/OS by rectang on Wednesday September 06, 2017 @05:00PM (#15187097)
Mesos is an Apache project. (DC/OS is not.) Apache projects are required to be vendor-neutral. That's part of the covenant under which contributors participate.

The Apache governance model provides opportunities for competing vendors, so besides Mesosphere, the market is open for other players to provide Mesos support.

Re:Run Kubernetes on top of DC/OS by trengrj on Wednesday September 06, 2017 @05:49PM (#15187389)
Components aren't open source. Still waiting for the new hdfs package to be released see https://github.com/mesosphere/hdfs-deprecated/blob/master/RE...
Re:Run Kubernetes on top of DC/OS by tknaup on Wednesday September 06, 2017 @06:30PM (#15187659)
Re:Run Kubernetes on top of DC/OS by trengrj on Wednesday September 06, 2017 @06:54PM (#15187801)
Yes the package is available but as far as I can see not open source yet.
Re:Run Kubernetes on top of DC/OS by spahl on Wednesday September 06, 2017 @07:59PM (#15188119)
Re:Run Kubernetes on top of DC/OS by trengrj on Wednesday September 06, 2017 @11:33PM (#15188974)
Excellent. Thanks!
Re:Run Kubernetes on top of DC/OS by jacques_chester on Wednesday September 06, 2017 @05:31PM (#15187294)
Disclosure: I work for Pivotal, which competes with Mesosphere in this arena. Read with discretion. If shilling persists, consult your techcrunch.

With that out of the way: you should try Kubo.

Google & Pivotal have been working on Kubo[0] to make the management of Kubernetes easier by using BOSH as the deployment/update/repair system. You make deployments by editing a yaml file. Or, better yet, but having a tool edit the file for you as part of a structured CI/CD pipeline (I've seen both and I am a fan of the latter).

BOSH works at the IaaS layer. Insofar as you mean "spin up and down VMs", Kubo is well-suited for that exact problem. You also get a bunch of upgrade and self-healing stuff for free.

Pivotal is also adding a commercial offering based on Kubo, PKS[1], which has been built with Google[2] and VMWare[3]. You might not need it for your exact usecase; the sweetspot is Pivotal Cloud Foundry users who want a relatively seamless side-by-side integration between CF and k8s. There's a bunch of value-added features (Harbor, GCP service brokers), of course, plus the whole throat-to-choke thing people like to pay for.

The pricing model isn't set in stone, but I suspect it will settle in the vicinity of what you have in mind. Our DNA is charging per-instance for Cloud Foundry apps, rather than worrying about sockets or RAM or other metrics which only poorly correlate to user value.

Anyway, I can hook you up with Kubo or PKS folk. Email me: [email protected]

[0] https://github.com/cloudfoundry-incubator/kubo-deployment

[1] https://pivotal.io/pks

[2] https://www.blog.google/topics/google-cloud/vmware-and-pivot...

[3] https://blogs.vmware.com/cloudnative/2017/08/29/vmware-pivot...

Re:Run Kubernetes on top of DC/OS by jdc0589 on Friday September 08, 2017 @09:13AM (#15199887)
I'm gonna check kubo out, cause its REALLY relevant to me in the coming months. Can you comment any why kubo might be preferable to something like kops?

You are probably aware of this, but people are starting to get a little fatigued from amount of automated ops/deploy tooling out there for k8s. Don't get me wrong, this is an area that needs improvement so competition is totally necessary, but its starting to become a bit difficult to navigate the landscape, or even keep up with the best choices.

Re:Run Kubernetes on top of DC/OS by jacques_chester on Saturday September 09, 2017 @03:33PM (#15208779)
> Can you comment any why kubo might be preferable to something like kops?

I'm unqualified to give a fair comparison, as I'm only skim-the-website familiar with kops. BOSH comes up much more frequently in my work. Most teams working on Cloud Foundry use it in some way, even if only to manage Concourse.

The main advantage that BOSH has over any of the others is maturity and production experience with large, stateful, distributed systems (first release was in 2010). It got the original abstractions right in a way that the alternatives of the time didn't. Chef et al are pet-builders, they excel in wrangling a single server into the target state you want.

BOSH instead says: why do you care about single servers? You're building a distributed system. If components drift or break, replace them with a clean image which was built from source.

As an example of production use, at Pivotal we use it to manage PWS. The Cloudops team use BOSH to roll out changes to a system running 10s of thousands of apps from thousands of users and companies.

In general, unless we land on a bug in the code rolled out, nobody ever notices. When we do have a bug, we can roll it back pretty easily.

BOSH isn't constrained to deploying Kubernetes. It was originally developed for Cloud Foundry and since then people have packaged up all manner of systems for it. We support some ourselves. For example, we have releases for RabbitMQ, MySQL and so on.

These can get used by on-demand service brokers too. Say an app developer wants a private RabbitMQ cluster. They tell the service broker to create a service, it has BOSH setup and monitor a brand new cluster, when it's done the dev can bind it to their app with a single command. Bing bang boom, totally self-service services. Nobody needs to fill out a risk form, file a ticket or pester their inside connection in ops.

One last advantage for Kubo and BOSH generally is that Google has assigned fulltime Googlers to both of them in multiple locations, working in pairs alongside Pivots. We've also become closely engaged with Google's new CRE program and it's been a really great learning experience for us.

Re:Run Kubernetes on top of DC/OS by wmf on Wednesday September 06, 2017 @05:28PM (#15187266)
This makes sense if DC/OS is targeted exclusively at on-prem.