Yeh flaky tests generally creep up in a big service due to several issues. There are some best practices to avoid the tests that requires some discipline and good oversight! We wrote some stuff around it: https://www.flakybot.com/blog/five-causes-for-flaky-tests

This is by no means an exhaustive list, but our goal with FlakyBot is to get better at identifying root causes as we identify flakiness across the systems.