How much slower?

Not much slower than, say, Hyper-V or Virtualbox or other hypervisors that live alongside a general purpose desktop OS. Remember, they're using the same Intel chips and VT-d and such as anything else; as long as you have enough CPU power it's going to run fine.

IIRC the Docker implementation for OS X runs a tiny, carefully-configured Linux VM transparently in the background. This isn't that dissimilar from the Windows Subsystem for Linux from a user standpoint.

I've done plenty of Docker work on OS X; the only real weirdness is that you're running Linux in the containers and OS X outside. Eventually, though, before switching completely to a native Linux desktop, I moved deeper into the containers and would barely ever see the OS X command prompt any more.

Our build time on Linux container is 8 minutes, but it's a few hours on macOS using docker containers. I haven't dug into it but the hypervisor seems to be single threading the normally multicore compile.

Primarily disk access is a lot slower, even with Hyper-V. Eg. Compilation suffers from this. But tasks that don't use the disk too much shouldn't be that bad.

Measurably slower than a LXC container, but still much faster than a full VM.