> It's not correct to say Docker was designed "not" to do persistent data.

Actually...looking at a lot of things "missing from" or "difficult in" Docker can more or less be traced back to Docker's overriding philosophy of container runtimes. Things like not having an init process and optimizing user experience around single "Run" commands.

Having relatively recently done a container retrospective, following the path of:

OpenVZ -> LXC -> Docker -> now (CoreOS, Rancher, LXD, Mesos, etc.)

I can't help but feel as if some of the "limitations" of Docker were more or less designed.

I'm not saying it's a bad thing, being a Python enthusiast I see value in simplicity/having only one way of doing things...but I just wanted to point out that where you draw the line for what is considered "designed" starts to get blurry in areas of Docker's chosen implementation.

Docker was specifically designed to NOT impose a model of ephemeral containers. For example, by default 'docker run' preserves all data.

Might not be the True And Approved Docker Way, but who cares? it works well, and has been in production for a number of years now. No data loss ever.

You know that trick the post mentions about how you have to keep removing images to stop your machines running out of disk space? Yeah, removing images doesn't actually reclaim disk space on devicemapper (!!):

https://github.com/docker/docker/issues/3182

Oh, and devicemapper is the only natively supported storage backend on the AWS official AMI.

Any monkey can run a bunch of stateless systems. It's the stateful ones that have the worthwhile problems.

I suggest using something that works for you and your business instead. If the only thing that works is $SOLUTION, but you have these difficulties, then yes by any means look for help.

Edit: Ok, I feel compelled to extend my reply as I've been downvoted without comment, which is terrible.

Here's the thing: You're not forced to use Docker just because it's the next hyped thing. It is great and yes, I use it. However there is more than way of doing things and the only reason to choose one technology over another is if it's better for the business.

Anything else is just ignoring reality.

Edit: case and point

> Please resist commenting about being downvoted. It never does any good, and it makes boring reading.

> Please don't bait other users by inviting them to downvote you or announce that you expect to get downvoted.

And since it is such a frequent occurrence, would that not point to the fact that people do enjoy talking about being down voted?

Actually CoreOS produced pretty stable stuff: etcd, fleet. Actually with fleet/etcd combined with some cgroups i see barely a reason to use docker, I mean on Cloud's its actually another useless layer, at least for me, maybe not in Google scale.

Not just that, CoreOS is basically made to push systemd-nspawn as a container framework as hard as possible.

It seems that they started to use docker while it was changing a lot, I didn't have any issue in the past few months with docker, but I don't use it in production.

2) You are not running it in production. Poster is. All sorts of issues you don't care about on your dev box matter a lot in production.

IOW, your experience can best be summarized by "just trying out Docker in a totally different way and it works great!" Irrelevant to this post.

How many filesystems has the Linux world created in the last 20 years? Of those how many are rotting piles of tire fire?

Filesystems are hard to get right.

ZFS is the way forward. For cross compatability, for reliability, for stability, for lots of use cases.

Btrfs is another tire to throw on the burning pile of other filesystems that won't work out in Linux.

Let the down votes begin.

But Linux has some major architectural issues that eventually the Linux faithful will have to admit to.

Containers.

Filesystems.

Observability. Though bless him Brendan Gregg is trying his hardest to help here.

Portability.

And this is what is so painful for the rest of us to watch or deal with. This inability for the Linux folks to admit these glaring architectural problems. Problems they refuse to look outside their bubble to see how others solved these problems and adopt solutions that have already solved these problems. They just want to continue to bury their heads in NIH soil. And double down on trying to solve these issues, poorly, on their own without any awareness of how others have solved these issues. People outside Linux land just might have the right ideas on how to solve these problems!

But I expect no one in that camp to acknowledge this and down vote away.

Sigh.

Frustration, pain are more appropriate terms from my pov than inflammatory. But ymmv.

It's like "Come on folks! You can do better than this! Please! Just stop! Think! Please. I'm begging you."

I'm not entirely convinced we should settle on ZFS just yet. It's fantastic and quite possibly the best option right now but it has a few limitations:

- The Linux implentation seems to have issues with releasing memory from the ARC back to the system

- The only way to expand a zpool is to add a new vdev. Pools are essentially a RAID0 of vdevs so if a single vdev fails, your entire server fails. You can mirror or RAID within a vdev but this means that the reliability of a vdev is the reliability of your entire pool. The problem here is that you can't just add 1 or 2 new disks since adding a 1 or 2 disk vdev would be data-suicide. For smaller servers, this is silly.

BTRFS looked like it was getting there but it proved buggy and unreliable. Personally, I'm waiting for bcachefs: https://www.patreon.com/bcachefs

If instead I ran RAID6, I'd have 80% of the disk available and I could add disks in single disk increments.

I think ZFS makes great sense for businesses that can throw money at disks but for smaller businesses or home servers it's kinda bad.

It was kind of a pain to configure (albeit quite flexible) but it's been pretty nice overall, already survived 1 disk failure and a capacity upgrade (during which I had to resilver after every individual disk upgrade, which was time-consuming, but after the last disk got upgraded, the extra space finally showed up)

Disk is cheap. There's no reason to design like it's not.

Plus, disk may be cheap but servers to house it are not (the kind of servers you'd run in your house, I know you can get cheap SC846 off eBay).

I'm sure it's not an issue in enterprise environments, but I've personally been bit by ZFS's inability to shrink or rebalance pools (even offline) several times now through personal use.

Those sort of use cases need to be handled if ZFS is to become a more general use filesystem.

tires++

;-)

Alright, the suspense is killing me: what OS is the one that gets it right? I ask this to "the rest of us" as you put it.

I'm not sure what reality you live in where Linux users like, actively deny things like lacking an appropriate COW filesystem with low latency, or things like the fact it took us a decade to get only close to DTrace. Your own mind? 6 month Ubuntu users? Internet forums where you talk to other BSD users and nobody else?

> Let the down votes begin.

Don't worry, I'm sure people will oblige.

However XFS is perfectly good for storage. I've used it on 15pbs worth of fileservers with no particular nonsense.

I wouldn't touch BTRFS with a barge pole, not because of any inherent stability issues (although that is a big factor) its the utter nastyness of the tooling.

ZFS is a joy to setup, and its simple, logical and the man pages tell you useful things.

BTRFS, not so much

1. filesystem is corrupted, once again

2. try to repair it

3. oh right, the repair tool can not replay the journal

4. try to mount it

5. admit after 3 hours of nothing that the journal-replay code triggered on mount actually can really not deal with corruption

6. reboot the server to get the filesystem unstuck

7. rerun repair, this time throwing away the journal

8. look at the empty filesystem with everything in lost+found

9. restore from backup

The team I'm in only runs 1'400 servers, yet this happens regularly.

But, mostly it happens because the fileservers dont have UPS's (The pipeline tools are almost exclusively COW, and backups are tested many times a week.)

I think we paid for support on XFS from either redhat or SGI, but I can't remember, I left that place a year or so ago.

I've never had the balls to run ZFS on large(100tb+) arrays. Last time I tried the way slab handling was translated from solaris cause many problems (but that was more than 4 years ago. ) Plus the support is a bit odd, you either go with oracle(fuck that), or one of the openZFS lot.

To get the best performance/stability you ideally need to let ZFS do everything, instead of letting the enclosure do the raid 6 (4 * 14 disk raid 6 with 4 spares). This of course is a break from the norm and to be treated with utmost suspicion.

Have you seen the GPFS raid replacement? thats quite sexy.

I run ZFS at home, because it works, and has bitrot checking.

Just as long as it does't get anywhere near capacity.

What a useless comment. If you are running out of capacity adding disks to a zpool is incredibly mind numbingly easy.

"zfs add poolname mirror disk1 disk2"

Every filesystem has performance hit the floor when you consume all available capacity. This is not unique to ZFS but to well, reality.

Both get you ZFS and something lighter-weight than VMs. Need Linux? Bite the bullet and use VMs.

So if you have the skill FreeBSD. Otherwise SmartOS makes a killer setup for containers unmatched by anyone else.

Maybe not so important in the container context, but sooner or later, somewhere you need persistence, and tape offers certain persistence features you pretty much can't get elsewhere, especially at its media price points.

(Plus, if they got this wrong, when it's really not that hard to get right (I've done SCSI at this level before), I wonder what else they have.)

No killing yourself necessary, really ;)

Docker has been actively trying to become to household brand name for containers so it's difficult for me to give them a pass with your line of reasoning.

> Re Persistent Data: You are probably going to have a tough time if you try to do persistent data. Containers really excel at stateless tasks(or temporary state like caching). Certainly it's doable but for someone early in their adoption the proper toe in the water is probably the stateless business logic not the state layer.

Not a company of good taste in my book.

Pick one.