Flags set via config files require constant updates to said files, not a problem in a small team.
Work in a secure environment where config is controlled? You now need a process for controlling all these new flags.
Have hard dependencies on config (this flag MUST exist)? You tie your application to your config, both now need to be released at the same time.
Code clutter? Yes and it is avoidable. Also flags are not suitable to every type of feature so you're in a half way house anyway. Something which requires a DB change COULD be done via a flag but not worth the effort.
My experiences are mostly around CD pipelines that start breaking because of complexity. Feature flags might suit some application deployments and that's great. In everything I've worked on, they can be replaced by incremental (non-breaking) updates. In that situation anyone can do the release, no deep-feature knowledge required.
A personal example: we were developing a new insurance billing workflow. Since it was a huge change, and very risky, and not all health insurance companies make it easy to test, we rolled it out practitioner by practitioner or customer by customer.
We got to about 95% 'enable_improved_billing_flow=True'. The remaining 5% were turning into a huge project to migrate due to the specifics of their practice / their primary insurance companies / multitude of business reasons (those ~5% were an outsized percentage of our revenue). In the end, we ended up having to maintain two billing flows.