It's slightly better now but I've done iOS+Android mobile projects before where CI for Android has been easy to set up and CI for iOS has been a complete nightmare.
Don't CI your apps, CI your libraries.
> Don't CI your apps, CI your libraries.
That's not scalable for complicated apps that you can write automated UI tests for though.
To me, what you described is an app that was not designed to be easy to test component-by-component & will have a high overhead on maintenance. Apple’s restriction on virtualising MacOS seems unrelated to how the app was architected, so it feels unfair to expect Apple to alter their position to better support something they weren’t responsible for.
We have complex commercial apps with fully automated user interface tests (e.g. it'll test that you can enter your username + password, actually login and you'll be able to see content). You could test all the individual components as much as you want but you're still going to get bugs at the UI layer that you can automatically test for.
We do the same for Android and CI for that is so much simpler.
> Apple’s restriction on virtualising MacOS seems unrelated to how the app was architected, so it feels unfair to expect Apple to alter their position to better support something they weren’t responsible for.
You'd think they'd want to support workflows that led to better quality apps being created. Surely it's their responsibility to support developers?
Surely you can prove the correctness with integration tests of your modules, and then your acceptance tests can be simple 'did the thing not show an error when we clicked on the thing, and it went to the right page'? We have 'simulate the world' 'acceptance' tests at work and they're terribly flaky, and I look back at the layered approach to testing we used in a previous job where the go-live test was to hit the landing page & search results page & check that the status code was 200.
To be fair, the kind of tests I'm proposing won't catch layout issues. But, neither will the tests you're using.
What made your UI tests so flaky? As long as the code for them isn't much beyond "enter text into text input X, click this button, you should be able to see this text" they're not too bad to write but the initial setup can be tiring.
> To be fair, the kind of tests I'm proposing won't catch layout issues.
Screenshot comparison based tools can be amazing for this as long as the way your app looks is fairly stable.
Automated testing is always a tradeoff with effort to setup vs the time you save (I'm against e.g. TDD for everything) but for complex apps and big teams it eventually pays off.
Overall we are happy with it. I don't think CircleCI 2.0 is available yet for iOS, though, so nothing new to discuss yet.
On the topic of QA testing/automation, it really kills me that Apple servers won't sign older iOS releases which means QA teams need to protect stock devices from accidental upgrades. The AWS Device Farm iOS devices are all jailbroken.
> (iii) to install, use and run up to two (2) additional copies or instances of the Apple Software within virtual operating system environments on each Mac Computer you own or control that is already running the Apple Software, for purposes of: (a) software development; (b) testing during software development; (c) using macOS Server; or (d) personal, non-commercial use.
[ed: i guess you mean running os x server is an alternative to "non commercial use" - I agree with that reading. The option for nc use seems like a nod to not completely make experimentation and creative development/research entirely illegal.]