The best example I’ve come across is building software from source. When there’s no package, you have to do this. Some may argue that this is not Ansible’s job, but I don’t see how it’s different from `apt install`, nor where else it would fit into the pipeline. Anyway, the way I solved this for example was by pinning the idempotency on the existence of build artefacts. Those are often software-specific, so it’s quite fiddly/non-general to find the right artefact.
Package it as a deb as part of the CI in say jenkins and then apt install it with ansible.
https://github.com/wtsi-hgi/hgi-systems/blob/master/ansible/...
The replies to my original post say this is not something Ansible should be doing. Fair enough :P I was young!
Basically, it's better to stick with tasks that stand on their own. As soon as you stick complex task logic in it becomes kind of a Rube Goldberg machine. That being said, you can pull off quite a bit if you just test rigorously and write defensively.
http://reclass.pantsfullofunix.net/index.html helped a lit but still not perfect
You can break it by app, like, having many git repos one per app and then use ansible galaxy to reuse code
Or you can try having a dynamic inventory that pulls metadata from say consul.
We’re in the ballpark of 2k machines and still happy !