For years, I advocated for developing inside of VMs or containers. Even wrote a very handy shell tool for managing the process. But a few years ago, I stopped, and just switched to developing on my local machine.

Working in VMs or containers adds a ton of complexity, for very little benefit.

Installing databases is trivial, with any of `brew`, `yum`, or `apt-get`.

Your `bin/setup` script can take care of automating that for onboarding new developers. The same script gets used in your Dockerfile.

And your CI system is there to as-perfectly-as-possible replicate production, to run your comprehensive test suite (you do practice TDD, right?) and catch things like "forgot to add a library dependency to the setup script" and "app broke because of a library version difference".

Since switching to local-only development, plus containers and CI/CD, my life has gotten a lot nicer.

At GitLab all team members develop without virtualization. Disk IO is just much faster that way. We recommend the approach in our development kit https://gitlab.com/gitlab-org/gitlab-development-kit

We would love to see better virtualization disk performance so it is easier for non-team members to contribute. We're happy that despite that people are still able to contribute https://gitlab.com/gitlab-org/gitlab-ce/merge_requests?scope...

That's interesting, but shouldn't containers (e.g.: docker/lxc) be almost as fast as native at disk I/O?

I mean that's basically why I moved my self hosted virtual servers from KVM, which was abysmal on regular desktop spinning disks, to LXC/Docker which is comparably fast to native -- plus I can fit a lot more containers than full VMs.

The problem is that on MacOS you have to run containers in a virtual machine since you're not running on Linux. Crossing the VM boundary with IO can be slow, especially on virtualbox.

Oh, I'm not using MacOS so I didn't know that.

It makes a lot more sense now, thanks!

If you are developing within a docker container, how do you get as far as CI before you have tested it working locally??

The whole point of using docker containers is to have the same development experience locally, in CI and in prod.. not have some unknown system libraries on your box which don't match other environments.

Want a database? docker run mysql/mysql-server

Your argument for switching away from containers, seems to actually be arguing for using containers.

You'll never get the same environment locally as you do in prod. That's why you have staging (you do have staging, right?).

It's better to make the environment as close as possible to prod without adding massive inconvenience but docker both adds inconvenience and forces you to use it in production (which entails a whole other set of headaches) if you want close environmental parity, and even then, since it's lightweight virtualization there are hundreds of things which can behave differently between dev and prod.

IMHO you either want very close environmental parity (in which case full virtualization is the way to go) or you don't, in which case running locally is fine.

> That's why you have staging (you do have staging, right?).

Plenty of people don't. The main reason being that staging is never quite the same as production...

We are working in a remote team, so replicating shared envs locally is important. Our system contains multiple databases, application servers, SMTP, etc etc. -- you can use all of them in your local dev env. It's not the same as prod because of different latencies, etc., but all containers are present locally. It is _very_ useful.

>We are working in a remote team, so replicating shared envs locally is important.

It is, but I'm not particularly keen on docker as a solution to this problem. It provides a low level of isolation/realism for these services and the tooling surrounding it is relatively poor.

What about LXC system containers? Differences between hardware and OS virtualization still exist, e.g. immutable system clock, but it's otherwise like running a regular Linux box. An init process that can simulate rebooting, mature system package management, no process isolation that's often gratuitous, etc.

What about LXC system containers? Differences between hardware and OS virtualization still exist, e.g. immutable system clock, but it's otherwise like running a regular Linux box. An init process that can simulate rebooting, mature system package management, no process isolation that's often gratuitous, etc.

> Installing databases is trivial, with any of `brew`, `yum`, or `apt-get`.

yes. For once database version and one single data set. If you handle multiple customers, multiple versions of a software suite to support which target specific versions of a DB, containers or VMs are handy.

I don't do my dev in a container. That always seemed stupid for me. But my dependencies are in containers.

This makes Docker for Mac unusable for rails development (without 3rd party work arounds). The issue and the radio silence has burned a lot of goodwill I and I'm sure others had for the Docker project. It's made worse by the constant stream of "production ready" announcements, while this product is crippled for most common use cases.

Here's the related thread on the docker forums. https://forums.docker.com/t/file-access-in-mounted-volumes-e...

I have given up on using docker on macOS. I now use Ubuntu at work, and it's interesting to watch a lot of my co-workers give up on macOS too. I totally agree it is a horrible experience.

Use Docker on Ubuntu on my own machine, but forced to use a Docker on Windows at work. Using Docker on Linux is an absolute joy. I totally advocate it. Dockerhub is my first port of call when I'm installing something new on my machine because I know if there's an official repo it will work.

Docker outside of Linux is another story. I've run into config issues, VM memory problems, proxy issues, ect. Painful.

Same here. I was struggling using docker at work on my Mac, but when I was working at home on my Linux box it was amazing. Docker is awesome on Linux and I have never been more productive.

This is the case where I work too. I'm guessing that at least 75% of our engineering team now uses Linux, even though everyone were given a MBP when hired.

Working with Docker at a previous gig, after wrestling with docker-for-mac and boot2docker for months I just stopped witl all the hacky work-arounds and just used Vagrant or Ubuntu AWS nodes for Docker development. Also, I have to say that Docker ONLY got a 'prune' option with 1.13 is amazing - I had aliases to do that years ago and was always a huge pain point! Why wasn't this rolled in last year?

Same with our team. Linux, for the most part, is a much better development experience. The only con is our front-end guy can't use Photoshop.

I don't even miss iterm2 anymore. i3 tiling WM is iterm2 for the whole desktop on steroids.

Docker for mac mounts your whole FS to the VM which is so fucking scary and dangerous. I posted about this somewhere in the github issues, but they just shrugged and carried on. I also asked if they could compile the kernel with btrfs support, and the same happened.

They should just open source the whole thing so people could help fix issues or understand how and why it works.

It's likely impossible to make volumes shared from the host to the container fast for all use cases.

It's also likely impossible to make them work as expected on Windows where the host volume is NTFS and the container is Linux.

So don't use shared volumes for code.

At Convox we offer a Docker development environment in the "convox start" command.

We manage syncing code into the container by watching the host file system and periodically doing a "docker cp" to get source changes into the container.

It works great and shows the power of the Docker API for taking control of your containers.

A bit more info is available here: https://convox.com/guide/reloading/

I've been using https://github.com/codekitchen/dinghy for a while...it transparently uses NFS for sharing and disk access is really fast

It's no coincidence that some projects such as Nomad and Confluent include statements such as "Docker For Mac Caveats", "Don’t use Docker for Mac" in their docs. https://www.nomadproject.io/docs/drivers/docker.html

http://docs.confluent.io/3.0.1/cp-docker-images/docs/intro.h...

While the installation experience of Docker for Mac is very good, the unfortunate downfall of the project is they try to hide what is really going on. You are running a VM and through a series of very clever tricks and fancy technology they attempt to make the VM hidden as though you are running Docker local (like in Linux). This has the side effect of odd behaviors. As a result in my own projects we struggle to support Docker for Mac also.

Yeah, this is my favourite: screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty Oh you can just login as passwordless root to the VM, then you know, access the mounted fs which is your root fs. Insane.

Are you referring to this?

  / # uname -a
  Linux moby 4.9.4-moby #1 SMP Wed Jan 18 17:04:43 UTC 2017 x86_64 Linux

  / # mount | grep osxfs
  osxfs on /Users type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
  osxfs on /Volumes type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
  osxfs on /tmp type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
  osxfs on /private type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
  osxfs on /host_docker_app type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)

Yes, it mounts you HD in /Volumes.

As far as I remember, doesn't Docker for Mac create a VM and then run the docker containers in the VM? That'd explain the terrible filesystem performance.

But that is not the issue. I think the speed of docker under the os hypervisor is fine. The bug is in in osxfs https://github.com/docker/for-mac/issues/77

It does, it's a lighter VM than VirtualBox which is what docker machine used, but still a VM nonetheless.

docker-toolbox you are talking about. The new version (Docker for Mac) with this problem doesn't use a VM anymore, but native virtualization on OS X. The old version doesn't have this problem.

Native virtualization is a VM...

xhyve spins Alpine Linux VM inside which it runs Docker Engine. So basically it don't deviate too much from running Docker prev. to docker-toolbox for macOS.

Yes, and especially for tools like PostgreSQL which uses features of the Linux filesystems which Docker simply didn't implement or implemented in a buggy way. My only solution was to use data volumes for PGSQL mounts, which at least provide native ext4 experience and compatibility but then you loose the directory sharing part.

What was your use case where you needed pgdata in both the HOST and a container (or multiple)?

Simply just backup it up. There is no way to back up Docker volumes, except for making an other container only for this purpose which runs rsync or something similar.

Docker for Windows works great of you want to try it.

Except that you can't use virtualbox anymore when hyper-v ist active... That's why I switched back to docker-machine on my windows pc.

boot2docker in a virtualbox actually seems to have much faster file system access. Slightly more awkward experience overall, but perf seems better

And then you didn't mention the whole concept of storing everything in one-huge-always-expanding-blob ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 file which contains all Docker related storage, images, data volumes etc.

Now that file just loves growing to 40 GB+ and there is no supported way to reclaim that space.

You can delete the file (or click Reset to factory defaults in the GUI), but then you loose all your Docker's storage, like data volumes.

> and there is no supported way to reclaim that space

That's one of the things addressed in the latest stable release. It now reclaims space on startup.

This has been one of the main reasons why I have not adopted Docker yet.

I develop on a Macbook and push things to Linux Servers.