Sounds like you are mixing release & QA, makes a lot of sense in a small shop.

In a larger team you’d have a small agent on each of your N (likely >> 3) machines, CI pushes to the agent for build/release/automated tests.

Then if those pass on a given system, it fires off a message to start manual QA validation for performance and other “intangibles”.

Why don't you use virtual machines?

Good question. 1) A MacOS virtual machine is probably illegal. 2) I need to test the builds on all three machines, and VMs don't handle OpenGL, USB drivers (e.g. for MIDI controllers), and audio drivers very well or at all. 3) I need to have a rough idea of performance of the build. VMs affect performance by non-constant/unpredictable factors. For example, CPU performance might be 1x the speed of a bare-metal OS, but graphics performance might be 0.25x. 4) I don't know how to write a shell script on a host computer that launches Windows 10 and runs commands with an MSYS2 Mingw64 shell.

> A MacOS virtual machine is probably illegal.

true. And that is a very sad state of affairs. I use the travis osx hosts for that, but it's not ideal. There's no interface, but at least you can check that the code compiles, runs, and passes automated tests. That's already huge!

For linux and windows hosts, it seems to me that it is a solved problem, as pointed elsewhere.

> A MacOS virtual machine is probably illegal.

It's not, as long as you run it on genuine Apple hardware. We use a bunch of macOS VMs that are 100% legal.

The problem with macOS VMs is that there are a lot of compatibility issues, and in my experience it's a lot of effort to set up macOS VMs. If you have the space and the money, real machines are a lot easier to deal with.

> I don't know how to write a shell script on a host computer that launches Windows 10 and runs commands with an MSYS2 Mingw64 shell.

Launching a VM via shell script is trivial. And you can install OpenSSH on Windows 10: https://github.com/PowerShell/Win32-OpenSSH

We use a combination of real machines and VMs for building software, and dealing with macOS VMs is a major hassle. There are always compatibility issues that are hard to debug (eg. don't try to configure your VM to use more than 2 vCPUs if you want to run macOS 10.12). There is a lot of manual work that goes into maintaining VMs, and often automating those tasks doesn't pay off. If your app has any GUI components, those are often buggy in VMs (anything using the GPU is extremely unreliable).

Our main build machine is an actual Mac, because it's just so much easier to keep it running.

VMs are nice if you need a lot of different setups (eg. one app we distribute has components that need to be built on different versions of macOS, and VMs are nice for that).