Also, considering I make exactly $0 per month from it, it's either be patient or use another service that had more capacity and faster builders.
I believe that Travis-CI had a higher limit for parallel jobs, but your builds would most often end up queued because of a low capacity for a certain type of agents. Also, something quite noticeable was their lower limits on build time, which is why for some builds a cache was necessary. It got a bit hairy when you needed a full successful build to properly populate the cache but they would never finish in time. And of course, the biggest drawback from Travis-CI back then was the lack of Windows support.
As for Azure CI, I believe there's something like a 10 parallel builds limit. And I've got exactly 10 configurations!
So sure, I'm missing caching that I've had on Travis. But honestly, it builds reliably fast anyway. On Travis, I always needed to wait a LONG time to get macOS agents, on Appveyor, we were limited to 2 parallel builds. Here, everything starts all at once, completes under 10 minutes (there's quite a lot to build and test).
If I could, I would love to have an sccache ( https://github.com/mozilla/sccache ) compatible distributed free caching solution for my C++ projects. That'd be a killer feature!