I think the TLDR summary is that MacOS doesn't work very well in the datacenter. The hardware is not designed for rack mounting. The OS licensing is not friendly to running VMs. I wonder if this is going to bite Apple in the long run. Building datacenters that run Linux is so much easier and cheaper. Microsoft has been trying to up their game but Apple apparently has no interest or no clue. Maybe that's fine. They are awash in money from their iPhone hardware so why should they care.

Edit: photo of goofy setup here https://www.macstadium.com/datacenters/

I'm not sure goofy is the word. Looks pretty neat to me. (recent tweet by their VP with a picture:https://twitter.com/brianstucki/status/897076500049969152)

But you're right, it needs to scale much better. I'm surprised how many people think macOS VMs are available at Amazon only to visit and see that's not the case. Maybe this is part of the reason for all the new Apple data centers...someday.

The OS licensing is friendly to running VMs as long as the underlying hardware is Mac hardware.

Building datacenters that run Linux is easier and cheaper, but Apple is a hardware company, they want to sell you hardware, so there's not much incentive for them to allow people to virtualize macOS on non-Mac hardware.

I wouldn't call a limit to two VMs friendly. Plus Apple doesn't make suitable hardware for this.

Huh, I didn't realize they limited it to 2 VMs, but I just checked and you're right.

That said, since they don't make appropriate hardware for this, I'm not sure I'd want to tax a Mac mini with more than 2 VMs anyway ;)

Shameless plug for open source C/C++ projects: if you add support for the build2[1] build system, we will CI[2] your project on Mac OS (as well as Linux, Windows, and FreeBSD with various versions of GCC, Clang, and VC) for free and without any limits.

For those curious about the setup, we have our own infrastructure that runs a custom in-memory Linux-based OS called Build OS[3] that performs build tasks in throw-away VMs (KVM). This is used for all targets, including Mac OS (though we have to run Mac OS builds on a bunch of Mac Mini's due to Apple's licensing restrictions).

[1] https://build2.org

[2] https://cppget.org/?builds

[3] https://git.build2.org/cgit/buildos/tree/

I build travis CI jobs on Mac OS because that's the easiest way to get a reasonably modern c++ compiler. I'm glad somebody realizes there's life beyond c++-03

Why can't you just run Clang on Linux? Surely its C++11 and later features aren't exclusive to macOS? Or are you talking about some other compiler?

Travis is still on Ubuntu 14.04, but that ships with clang 3.4 and gcc 4.8, both of which support C++11 just fine, so yeah, I'm confused why 'ksherlock needs macOS. clang 3.4 even supports C++14 (gcc needs version 5; also you might want gcc 5 for the C++11 ABI changes to std::string and std::list).

'ksherlock, if you really want a newer compiler, I'd use http://apt.llvm.org/ . Specifically do something like

    sudo: required
    
    before_install:
        - curl https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
        - echo "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty main" | sudo tee /etc/apt/sources.list.d/llvm.list
        - sudo apt-get update
        - sudo apt-get install clang-6.0

which will get you the latest nightly build of the not-yet-released clang 6.

Alternatively I think you can do it like this without sudo: required:

    addons:
        apt:
            sources:
                - llvm-toolchain-trusty-5.0
            packages:
                - clang-5.0

but clang-6.0 isn't on their whitelist https://github.com/travis-ci/apt-package-whitelist . Maybe send in a PR!

Apple can't be unaware that this is a problem, right? They have to be one of the biggest users of such a solution for building and testing software on Macs and iOS devices. What do they do? And why aren't they (either themselves or in partnership with someone like IBM) offering a SaaS product for Mac/iOS developers to handle this sort of thing?

It is possible to run macOS on beefier hardware (if you ignore the license) and run more VMs per host (again, if you ignore the license). In Apple's case, they don't have license issues to worry about and thus can do their SaaS/CI stuff in VMs on as many of the beefiest servers that make sense to buy instead of having to fight with Mini's or racking Mac Pros. I expect that most likely they are currently doing that on beefy machines from a cloud hardware vendor like Quantas or Dell.

Sure, but Apple has to know that about their license, right? So why haven't they (either directly or with a cloud vendor) provided a SaaS product like this?

Is there any service like Travis CI or CircleDI that lets you add your own slaves? Jenkins let's you turn a box into a slave by running a simple jar file.

It would be nice if CircleCI or Travis CI offered as that would mean not having to pay them for instances and instead turning AWS instances into CircleCI/Travis slaves at a lower per-instance cost. I'd still be happy to pay for their website and for their system managing the slaves.

GitLab CI does this.

It has always been avoided in Jenkins-based systems for security purposes. Traditionally, slaves could request that masters do pretty much anything, as Jenkins treated the master/slave relationship as one large system rather than isolated and restricted nodes.

This has gotten slightly better over time, but the default is still "slave is just another process alongside the master" – you need to setup any whitelisting of slave filesystem or execution permissions yourself.

Hosted service: https://buildkite.com

Self hosted: Gitlab

To be clear, you can do this with hosted GitLab.com as well.

We use Travis for Mac & Linux builds of OSS projects (e.g., https://www.telepresence.io). What's frustrating is that even the paid service for Mac is just as bad. We're on the East Coast, so we can use Travis for Mac in the morning, but it's terrible by lunchtime.

Fun fact: before Travis had multi-stage builds, we were able to get multi-stage builds in Travis by using the fact that mac builds were ALWAYS slower than Linux. So we could have our Linux builds create artifacts that were then used by Mac builds ...