News for hackers. Stuff that matters.
Re:CoreOS Stable Release by rckclmbr on Friday July 25, 2014 @02:40PM (#8086829)
We recently switched (like 2 weeks ago) our project from deployment on ubuntu servers via 'git pull' managed with supervisord to docker/coreos/fleet, and it's been epic. While coreos is built for large clusters, we run a 3 host cluster in ec2, and couldn't be happier. We switched from multiple servers running 1 instance of each service to load balancing all instances on these 3 hosts. This increased uptime, made deployment and management easier, and gave us the benefits of docker as well (verifying things work locally).

There's only 2 real problems, both of them very minor:

* fleet managing state. We've had to manually kill containers sometimes, and destroy systemd services before we could start it again.

* all EC2 amis use ebs backed instances. We haven't used a higher-IOPs ebs backed instance because the only delay we see are in startup times (which doesn't matter, just longer rolling deploys). But an instance-backed ami would be nice.

Re:CoreOS Stable Release by bcwaldon on Friday July 25, 2014 @02:57PM (#8086951)
We're actively working on fixing the fleet state management problems. It's been a pain for us, too.

https://github.com/coreos/fleet/pull/638

Re:CoreOS Stable Release by superuser2 on Friday July 25, 2014 @02:54PM (#8086926)
How do you handle persistent storage in a fault-tolerant way?

You could technically pin your MySQL container to one host, but that seems to defeat the point of fleet. I considered trying to mount an iSCSI target to run the database from, but CoreOS doesn't have a working iSCSI initiator.

I guess I could just run all the persistent stuff on a more traditional OS, but then why mess with CoreOS at all?

Re:CoreOS Stable Release by leg100 on Saturday July 26, 2014 @05:00AM (#8089264)
The persistence can follow the DB container:

https://github.com/leg100/docker-ebs-attach

You'll need three fleet units in total:

  - one to attach the EBS volume (wrapping the above container)
  - one to mount the device to a directory on the host
  - one to run the DB container, with a bind mount to the host directory
If the CoreOS host terminates, the cluster will reschedule the units to another host.
Re:CoreOS Stable Release by rckclmbr on Friday July 25, 2014 @02:57PM (#8086957)
We use Amazon RDS and S3. Only applications are run in the cluster.
Re:CoreOS Stable Release by superuser2 on Friday July 25, 2014 @03:06PM (#8087024)
Yeah, this seems to be common.

CoreOS seems to do a decent job as the application layer for something operating in a larger cloud that provides database and file storage as services, like Amazon or OpenStack. However, I think Docker has the potential to eliminate the need for infrastructure-level "clouds" to be so complex. The basic unit can be the container instead of the instance, so you don't need fancy automated instance management and all that. It'd be nice if you could also obviate the need for the other external services, and just run everything in containers on metal.

I was really excited by CoreOS as a way to do this, but it seems to fighting me at every turn. It seems much easer to just wire together the REST APIs of several Ubuntu/CentOS nodes running Docker.

Re:CoreOS Stable Release by opendais on Friday July 25, 2014 @06:03PM (#8087899)
I haven't used CoreOS, mainly just watched, but have you tried deploying a Galera cluster at a scale of 5 containers?

That is more likely what they'd have in mind than a single MySQL master container.

Re:CoreOS Stable Release by otterley on Friday July 25, 2014 @06:26PM (#8088000)
Who uses Galera successfully? Every DBA I've ever spoken to about it has advised me to avoid it because it's incredibly deadlock-prone.
Re:CoreOS Stable Release by opendais on Friday July 25, 2014 @07:52PM (#8088288)
http://blog.scoutapp.com/articles/2013/09/24/from-mysql-mmm-...

That explains how they chose to minimize deadlocks.

Yes, if you use it incorrectly and/or for the wrong problem space, you will have issues. It isn't a magical solution to all problems. It is, however, an effective solution for a specific problem space.

Re:CoreOS Stable Release by kraig on Friday July 25, 2014 @07:52PM (#8088291)
I've been using it in our development environment. It locks like crazy.

This is after configuring HAProxy to have a primary node which all connections go to.

OpenStack seems to be OK with it though, it chugs along without fully breaking down. I'm sure I would see a pretty big performance delta if I ran a benchmark against it with and without Galera.

Re:CoreOS Stable Release by tryphan on Friday July 25, 2014 @07:23PM (#8088191)
I'd like an answer to this as well. Openstack installation guides recommend it for HA.
Re:CoreOS Stable Release by LunaSea on Friday July 25, 2014 @08:00PM (#8088308)
What OS image do you use for Docker ?

Is it still Ubuntu or do you use CoreOS on both the "host" and "base image" ?

Re:CoreOS Stable Release by cyounkins on Friday July 25, 2014 @11:24PM (#8088824)
Using CoreOS as a "base image" under docker would serve little purpose because CoreOS isn't really designed to run anything other than its core services. People usually use Ubuntu or CentOS.
Re:CoreOS Stable Release by rspeer on Saturday July 26, 2014 @01:18PM (#8090193)
Wait, little purpose? Isn't that the main purpose? "Applications on CoreOS run as Docker containers." [1]

Am I misunderstanding what CoreOS is for or are you?

[1] https://coreos.com/

Re:CoreOS Stable Release by robszumski on Saturday July 26, 2014 @03:37PM (#8090535)
He's saying that there isn't a point of running CoreOS as the base image inside of a container, which is true for most people. There isn't a good reason to run CoreOS containers on a CoreOS host.
Re:CoreOS Stable Release by rspeer on Saturday July 26, 2014 @05:40PM (#8090892)
Oh! I had a different sense of "base ... under Docker" in mind.

My interpretation was that Docker was running on top of an OS, which you could call the "base", and say it was "under" Docker in the sense of "underlying".

Whereas he was using the opposite spatial metaphor: the Docker image, which is a "base" in the Docker sense because you derive other images from it, is "under" Docker in that it's within Docker, or lower in the process tree.

George Lakoff would be thrilled by this example of conflicting metaphors.

Re:CoreOS Stable Release by outside1234 on Friday July 25, 2014 @02:54PM (#8086934)
Have you written up this in a blog post anywhere? Would be an interesting read and very valuable for dev/test scenarios in addition to production ones.
Re:CoreOS Stable Release by rckclmbr on Friday July 25, 2014 @03:03PM (#8087003)
Not yet. I will shortly.
Re:CoreOS Stable Release by micah_chatt on Saturday July 26, 2014 @09:50AM (#8089731)
I'm really curious how you manage the load balancing. Are you using CoreOS's elb mailgun walk through?
Re:CoreOS Stable Release by rckclmbr on Saturday July 26, 2014 @01:25PM (#8090214)
Yep, exactly.