News for hackers. Stuff that matters.
Re:I Want to Run Stateful Containers, Too by jacques_chester on Sunday November 22, 2015 @09:30AM (#10610127)
I agree with the author that writing your own snowflake PaaS is a mistake. I work at Pivotal on the fringes of CloudFoundry; OpenShift Origin is a competing system. Either way, you should be using a full PaaS instead of rolling your own.

But this is the bit that surprises me:

    They get you hooked for free and the next level 
    is $1,496 per month… wtf! MongoLabs is little 
    better.
$1500 per month, versus weeks of engineering time spent tinkering with and upgrading and bug-fixing and trouble-shooting and security-patching a hand-made solution is a fantastic bargain.

Supposing the author was running on Pivotal Web Services. It would've taken less time to add a MongoLab service (about 2 minutes, 3 if you include a re-stage) than to perform the CloudFormation calculation (say, half an hour, resulting in no running software).

PaaSes like Heroku, Cloud Foundry and OpenShift are feature-complete for the cases that application engineers and operators care about. If you roll your own you're directing effort to something that doesn't provide user value.

If I walked in on someone at a regular dev company rolling their own operating system for a web service, I'd be surprised. Writing their own programming language? I'd be skeptical. Oh you built a new HTTP server? Why? Outside of a research environment, why are you doing that?

And so it is with PaaSes. The marker was passed years ago. We don't need to go on these spiritual quests any more.

Disclaimer: I work for Pivotal, which donates the majority of engineering effort on Cloud Foundry. I'm actually in Pivotal Labs, the agile consulting division, which is where I morphed into a lay preacher for just-using-a-PaaS-dammit.

Re:I Want to Run Stateful Containers, Too by FooBarWidget on Sunday November 22, 2015 @09:34AM (#10610143)
> $1500 per month, versus weeks of engineering time spent tinkering with and upgrading and bug-fixing and trouble-shooting and security-patching a hand-made solution is a fantastic bargain.

That's only true if you look at it from the point of view of an enterprise, a startup with VC funding, or generally a "rich" company. If you're a cash-strapped startup/small business with only a few thousand dollars profit per month, $1500/mo is a huge deal.

Re:I Want to Run Stateful Containers, Too by curun1r on Sunday November 22, 2015 @11:52AM (#10610583)
This is especially true when you consider what that same $1500/mo will buy you using something like DynamoDB or Aurora. Both those solutions will give you more storage, are managed for you and will mostly scale up with you, meaning you don't have to start anywhere near $1500/mo.

I know the article said tying yourself to Amazon feels wrong. But focusing your time and energy on managing infrastructure that could be managed by Amazon instead of focusing your time and energy on your product feels more wrong to me. There are exactly zero startups that have succeeded because they had a more reliable and performant MongoDB installation.

Re:I Want to Run Stateful Containers, Too by nzoschke on Sunday November 22, 2015 @01:34PM (#10610917)
A big +1 for DynamoDB + S3 for a cash strapped startup.

If you are writing an app from scratch, and the cost of data services is a big concern, then don't pick a data store with complex and expensive replication properties.

DynamoDB is admittedly harder to understand and use than Postgres or Mongo, but when you figure it out its a HTTP data API with no setup or maintenance costs.

$8/mo of DynamoDB can easily cover your users and other CRUD.

Re:I Want to Run Stateful Containers, Too by vacri on Sunday November 22, 2015 @02:54PM (#10611213)
> If you're a cash-strapped startup/small business with only a few thousand dollars profit per month, $1500/mo is a huge deal

If it takes 30+ hours of engineering time per month (less than one work week), then you've hit your break-even point. This is assuming the engineer is $50/hour, including overheads.

Re:I Want to Run Stateful Containers, Too by wbkang on Sunday November 22, 2015 @09:47AM (#10610182)
Wouldn't need to worry about auto scaling Mongo yourself if you at that stage of the company though? You'd need to have quite a bit of traffic to need it. And if you do, hopefully you have generated $ needed to support it.
Re:I Want to Run Stateful Containers, Too by jacques_chester on Sunday November 22, 2015 @11:39AM (#10610540)
Engineering time isn't free, even for cash-strapped startups.

For a startup, your hyperfocus has to be on creating a product that is attractive to users. Everything else is entirely secondary.

Re:I Want to Run Stateful Containers, Too by vinceguidry on Sunday November 22, 2015 @10:48AM (#10610379)
Yes, I think devs sometimes forget that this is the bleeding-est of bleeding edges. Hell, I can't even get a normal Digital Ocean server instance to programmatically drop and recreate doing minimal things like creating a user and setting SSH keys. I suppose if I wanted to learn something heavyweight like Chef or Puppet I could, but for now, I just maintain a list of commands to run, and understand that Digital Ocean servers can't quite be treated like livestock just yet, though I can fake it pretty well. Eventually Digital Ocean will get around to making the platform changes, which are all already in feature request.

Engineering time is very limited and the support cases are exponentially large. The author wanted MongoDB supported in the hard case. Well, that means you also need to support 4 or 5 other, similar databases the same way.

If you want a bleeding-edge PaaS with corporate support, you should expect to pay corporate prices. If the project / organization doesn't allow for that, then either the requirements or the mindset is off-kilter.

I think the what the author is really mad about is the vendor lock-in. That used to bother me too, until I realized that learning new platforms is done on my employer's dime and not mine.

A few more years, and some other challenge will be on the bleeding edge and this one will be mostly solved, meaning that open-source community solutions will exist that you can build once and forget about it. Knowing most devs, they'll spend all their time complaining about whatever the new edge is instead of marveling at how much easier things are now compared to the 'bad old days' of 2015.

Re:I Want to Run Stateful Containers, Too by m-app on Sunday November 22, 2015 @04:16PM (#10611524)
> Hell, I can't even get a normal Digital Ocean server instance to programmatically drop and recreate doing minimal things like creating a user and setting SSH keys. I suppose if I wanted to learn something heavyweight like Chef or Puppet I could, but for now, I just maintain a list of commands to run, and understand that Digital Ocean servers can't quite be treated like livestock just yet, though I can fake it pretty well. Eventually Digital Ocean will get around to making the platform changes, which are all already in feature request.

It sounds like you might benefit from Cloud-config for droplets: https://www.digitalocean.com/community/tutorials/an-introduc...

Or have you tried that already? It's also available through their API, BTW.

Re:I Want to Run Stateful Containers, Too by vidarh on Sunday November 22, 2015 @11:03AM (#10610423)
> And so it is with PaaSes. The marker was passed years ago. We don't need to go on these spiritual quests any more.

The CloudFoundry installation instructions alone makes me run away screaming. Same with OpenStack.

There's no "spiritual quest" involved, but a lack of willingness to add unnecessary complexity. Of course it looks different when you've already eaten the cost of getting it up and running and/or has readily available in-house expertise.

Re:I Want to Run Stateful Containers, Too by jacques_chester on Sunday November 22, 2015 @11:50AM (#10610569)
> The CloudFoundry installation instructions alone makes me run away screaming.

BOSH is getting some much-needed love right now for this reason. Incidentally, for long-running stateful services, BOSH is the right tool for the job. We use it for Cloud Foundry, MySQL and a bunch of others I don't recall right now.

Anyhow. If you just want to see quickly if CF fits your needs, the place to go is Lattice[1], explicitly designed to be installable on a laptop.

In the meantime, you can let Pivotal host you on Pivotal Web Services[2].

That plucky startup from Armonk, IBM, are the second-largest donor of engineering effort to Cloud Foundry. Their public installation is BlueMix[3].

Other hosted installations of Cloud Foundry are CenturyLink AppFog[4], anynines[5] and Swisscom Application Cloud[6].

When you decide you want something in-house, you can buy Pivotal Cloud Foundry with all the support bells and whistles. Or you can just use the standard cf-release repo and deploy it to you own OpenStack or vSphere environment. Or on AWS. Or on Azure. Or some mixture of the above.

[1] http://lattice.cf

[2] https://run.pivotal.io

[3] https://console.ng.bluemix.net/

[4] https://www.ctl.io/appfog/

[5] http://www.anynines.com/

[6] https://www.swisscom.ch/en/business/enterprise/offer/cloud-d...

Edit: I'd be interested in hearing why people object to this comment.

Re:I Want to Run Stateful Containers, Too by vidarh on Monday November 23, 2015 @09:30AM (#10614739)
> Anyhow. If you just want to see quickly if CF fits your needs, the place to go is Lattice[1], explicitly designed to be installable on a laptop.

The issue is not whether or not CF can meet my needs, but that it is crazily complex. That it is complex enough that there's a need for a separat setup "explicily designed to be installable on a laptop" is to me an admission that CF is flawed from the outset. I don't care if it's' installable on a laptop, incidentally, I do all my development and testing and experimentations on setups that matches (in architecture anyway) the environments I'll deploy to. Exactly because I refuse to follow a setup built to be easy to get up on a dev machine only to find out that the full production deployment ends up being totally different (a lot of software fails here).

> In the meantime, you can let Pivotal host you on Pivotal Web Services[2].

No, I can't. It's not remotely cost effective for the types of things I do. E.g. for one client I currently part-time manage a private cloud environment with abou 150 containers. Firstly, none of Pivotals options offer enough CPU or memory for us to be able to run it on 150 of the 2GB container options (it'd be more like 500+ at that size), but even if 150 was enough, the estimated cost for that would be substantially more than what it costs to lease racks + power + bandwidth + leasing costs on all the hardware + charging the cost of hardware upgrades/maintainenance and the cost of server admin. With more realistic container counts, it'd bankrupt the company in question.

> and deploy it to you own OpenStack or vSphere environment. Or on AWS. Or on Azure. Or some mixture of the above.

The complexity and/or cost of these options makes CF massive unattractive for me. I'm just in the process of cutting a clients hosting costs by about 80% by ditching AWS for a setup on managed servers, for example, and the full deployment of all the infrastructure we need is substantially simpler than just deploying a "bare" OpenStack deployment, and certainly nothing like putting CF on top. They can afford an awful lot of my time to help implement additional functionality for what they save by that move.

Re:I Want to Run Stateful Containers, Too by parasubvert on Tuesday November 24, 2015 @02:27PM (#10622954)
"That it is complex enough that there's a need for a separat setup "explicily designed to be installable on a laptop" is to me an admission that CF is flawed from the outset."

I don't get this. The setup for a Chef or Puppet managed 1000 node setup is quite different than it is for your laptop. Is it somehow fundamentally flawed?

"Exactly because I refuse to follow a setup built to be easy to get up on a dev machine only to find out that the full production deployment ends up being totally different (a lot of software fails here)."

I agree. That's actually not the case in CF / Lattice land, the install is actually bit-identical, Lattice just removes stuff you may not need but a big organization might need (multi tenant security, access/identity management, infrastructure orchestration, etc.) Lattice is pretty dead simple to get up with Terraform or Vagrant. The actual pieces aren't particularly complicated relative to other container schedulers like K8S or Mesos+Marathon.

"I'm just in the process of cutting a clients hosting costs by about 80% by ditching AWS for a setup on managed servers, for example, and the full deployment of all the infrastructure we need is"

Right, I don't think a platform cloud like CF or even Kubernetes is appropriate to your client base's needs. My issue was with the concept that something like CF is "fundamentally flawed" because it doesn't meet your niche use case. It fills a different niche that presumes an infrastructure cloud to begin with. Most companies are moving OFF of managed services onto clouds, so it's a reasonable assumption. But there will always be a market to cost cut with a dedicated setup, where you play, and that's great. The market's big enough for everyone to get what they want.

Re:I Want to Run Stateful Containers, Too by thewoolleyman on Monday November 23, 2015 @03:15PM (#10617008)
> I'm just in the process of cutting a clients hosting costs by about 80% by ditching AWS for a setup on managed servers...

And what are the additional long-term development/sysadmin costs of having to manage those servers and infrastructure?

And how did the client factor in the risk of being tied to you and your knowledge of this particular managed server setup?

And how did the client factor in the risk of being tied to a particular vendor's managed servers?

It may very well have still been the right decision for the client to move from cloud to managed servers.

However, my point is that if you solely base the cloud vs. managed decision on hosting costs, you are ignoring other real long-term costs and risks that are not as readily apparent, but can still have a significant impact in terms of upgrades, maintainability, downtime, availability, changes in personnel, etc.

Re:I Want to Run Stateful Containers, Too by vidarh on Tuesday November 24, 2015 @06:36AM (#10620218)
Every time I've moved people off AWS, there's been an immediate and substantial and ongoing drop in development and sysadmin costs as we have far more flexibility in setting up an environment that actually fits that customer and can make guarantees about the system that makes for a less complex system overall.

As for risk of beng "tied to me", that risk is far greater with AWS - the number of "moving wheels" is vastly greater. If I was thinking primarily of staying busy, I can easily charge far more for my AWS work than for the more "mundane" setups, so from tht point of view it'd be in my interest to stick everything on AWS. The market rate and availability for "regular" ops guys that can handle these setups is far better - at least here in London.

In terms of being tied to a particular vendors managed services: They're not. Getting out of lockin is part of the appeal of getting off AWS once they first make that decision (and to be clear, if they insist on deploying on AWS, I do that too - ultimately it needs to be their decision). The setups I do for clients typically only require a method for bootstrapping a server to a pre-agreed basic Linux setup (usually CoreOS these days) and bringing up ssh with a key. Everything beyond that is generally easy to make provider independent. We do this exactly so that they can mix and match, sometimes at the same time (e.g. I have one client that is currently running systems at AWS, Google and a managed provider at the same time, deployed off the same little setup).

The do get dependencies on certain performance characteristics etc., but typically they can easily be met at dozens of providers.

> However, my point is that if you solely base the cloud vs. managed decision on hosting costs, you are ignoring other real long-term costs and risks that are not as readily apparent, but can still have a significant impact in terms of upgrades, maintainability, downtime, availability, changes in personnel, etc.

I agree you have to consider these too, but generally they tilt the numbers further in favour of managed servers or co-located servers.

If we were comparing with "old style" manually deploying applications straight to bare metal, I'd agree. But the more realistic comparison is to deploy to containers or VMs running on a thin OS layer on bare metal or a hypervisor. The point is not to get rid of the cloud abstractions, but to get the flexibility of being able to pierce the clouds so to speak and make decisions about the lower layers to keep cost and complexity under control.

The main reason I see for AWS growth is that in my experience most of the tech guys pushing for it have no visibility to costs and budgets. Even lots of middle management don't. And upper management rarely understands the technology tradeoffs.

I've been in the bizarre situations in the past of having people ask me what I could possibly need salary statistics for my development team for, because it didn't cross anyones minds to actually take out cost data to determine how much money we were actually allocating to ops vs. request features vs. bug fixing, or to determine metrics for e.g. when it makes sense to simply buy more server capacity vs. spending developer time to optimize.

The biggest problem with this is that people look at "cloud" as "I get to delete this capex line" without understanding the new opex that comes with it, and without understanding how little of the ops time a well run team actually spends on the small bits of low level hardware stuff that is unique to managing your own hardware vs. renting vms.

Most people simply don't understand the cost of the technology choices they are making.

Being able to close the gap for people there is a very valuable service (and here's career advice to anyone making the leap into technology management: Be that guy that can actually price out the technology solutions your team proposes and that can digest and make the technology costs understandable for the business guys - it makes you surprisingly rare)

There are certainly people for whom AWS is the right choice. E.g. if you can make extensive use of spot pricing and bring up huge number of instances for short periods, then it can be hard to beat. But most people just don't have thos usage patterns.

Re:I Want to Run Stateful Containers, Too by jacques_chester on Monday November 23, 2015 @07:17PM (#10618296)
We're just not going to agree on this. I hear you saying "I can roll my own for less", and in my head I'm seeing all the snowflake PaaSes I've seen in the past few years while working for Labs.

It looks good now.

Then you move on and someone else is stranded with a completely custom environment that only does the things you thought of at first.

Cloud Foundry isn't complex for a developer. That's the point. Deployment involves typing `cf push` and waiting a minute or two.

But the essential complexity of managing very large herds of heterogenous applications and services doesn't vanish simply because you aren't using them right now. It's always latent. We and other companies working on Cloud Foundry have already fixed problems you have never thought of.

Re:I Want to Run Stateful Containers, Too by vidarh on Tuesday November 24, 2015 @07:16AM (#10620299)
Who said anything about rolling my own? I do nothing of the sort. Most setups don't need one. Most needs a tiny selection of functionality that are easily served by selecting from an array of well tested, well understood, components that needs little enough glue that a single person can get the grasp of it a day or two. We know this, because we've gone through the exercise of bringing in people from the outside to walk through more than one system to validate the setups for various customers.

I understand your concern about maintainability, and that's exactly where I come at it from. I had developers on my teams in the past that would swear at me (one tried to have me fired) because I made technology choices they found boring and un-sexy because I actually paid attention to our long term costs, such as hiring replacements and ease of even finding replacements. I care deeply about that. And what I'm consistently seeing is that this definitively does not speak in favour of pulling in large, complicated systems like OpenStack or CloudFoundry unless you're working on very large, complicated systems that actually needs all the functionality they offer and where you can justify a team to learn how it hangs together.

Typically my requirement from the outset is that a mid level devops guy should be able to take over with minimal training for the simple reason that I have much more lucrative work to do than ongoing maintenance, so I'm 100% dependent on setting up systems that can be quickly and effectively managed by someone much cheaper than myself and that is part of the appeal.

As for not being complex "for a developer". I disagree. Automating a build pipeline is trivial compared to getting the application architecture right, and that is the area where dev teams often fall down, and I see all to often that they fall down because they don't realise what it is they've been signed up for and assume everything will be taken care off by magical ops fairies because they have way too little visibility into what is actually happening when they deploy something.

> But the essential complexity of managing very large herds of heterogenous applications and services

Most companies never end up managing "large herds of heterogenous applications and services".

And this may be the fundamental disconnect.

If you are dealing with "large herds", then by all means pick a comprehensive platform, as presumably you have the resources to manage it properly too. But the type of businesses who do are relatively speaking few and far between. I'm sure you see lots of them because of what you do, and I'm not saying there aren't good use cases for Cloud Foundry and similar for certain types of businesses.

What I am saying is that most companies don't ever even get to the size where they can afford that complexity, much less to the size here they need it.

Re:I Want to Run Stateful Containers, Too by bjterry on Sunday November 22, 2015 @02:52PM (#10611206)
It reads as an extended advertisement for you company's services, and your comment doesn't really respond to the statement of the person you replied to. He said CloudFoundry's installation instructions were bad, then you replied as if he said he was unsure if CF "fit his needs" and was looking for more ways to try it out, which is a pretty salesy response. I think mostly it's a question of tone and length. If you had a couple sentences acknowledging that the installation instructions sucked, and saying he should go with BOSH, you probably wouldn't have gotten downvoted.
Re:I Want to Run Stateful Containers, Too by jacques_chester on Sunday November 22, 2015 @04:43PM (#10611608)
Thanks. I was too emotionally invested.

I actually work for Pivotal Labs, not Pivotal Cloud Foundry (though as of this week I'm on secondment to a PCF-related team).

As a consulting engineer I get to see a lot of projects in a lot of companies. My eagerness for PaaSes comes from seeing a variety of approaches. Just using a PaaS makes large, expensive, disruptive discussions simply disappear.

Those who go through my history will note that I take pains to mention other PaaSes, usually Heroku and OpenShift. When I talk about hosted Cloud Foundry I usually namecheck ours and IBM's.

Re:I Want to Run Stateful Containers, Too by nzoschke on Sunday November 22, 2015 @03:15PM (#10611305)
I am working on Convox, an open source "PaaS" that installs in your AWS account in minutes.

There is no additional layer of complexity. It configures AWS coherently with a few cloud formation templates.

I'd love to hear your opinion on this type of installer.

http://convox.com

Re:I Want to Run Stateful Containers, Too by vidarh on Monday November 23, 2015 @09:36AM (#10614777)
It's nice if you are ok with being tied in to AWS, but I generally recommend my clients to stay away from AWS because of the massive cost premium. The cases where they've ignored that, I've generally been able to bill thousands to move them off AWS later when they've realised how much they're spending.

I certainly would never choose to build on top of another layer that's AWS dependent - if I'm first going to layer something on top of AWS, I'd use it as an opportunity reduce AWS dependency for the near inevitable moment when I'm asked how to move somewhere cheaper.

Re:I Want to Run Stateful Containers, Too by orf on Sunday November 22, 2015 @12:55PM (#10610789)
On what planet is it worth paying $20,000 per month for a vm??