We had built and authentication service that, among other things, was used by a SyncML service that was used back in the day of feature phones to syncs contacts etc. You can imagine that getting someone else's contacts on your phone isn't exactly ideal. This was how we came to know about the problem, from customers getting other customers data!

The error was caused by a CDN switch. Our instructions to the the CDN team responsible for the switch was "Make sure the CDN honors our cache headers, if our HTTP responses say something can be cached do so, if they say that the response should not be cached then don't". We were in at least three meetings where we repeated this mantra.

I believe that the CDN team thought that they had setup the CDN correctly but they had missed an edge case. The CDN was in fact setup to cache even uncacheable responses, and served those, _only_ when it could not reach our servers.

So if there was a traffic spike and the CDN determined that our authentication servers were unreachable it would fall back to serving data that should never have been cached in the first place! Happily returning tokens to random users that had authenticated just before the traffic spike...