Yup, don't forget the fact that you'll end up doing pooling, free lists and the exact same performance tricks you'd need to do if you managed memory explicitly.
You're adding a ton of incidental complexity for a benefit that you can't take advantage of(programmer productivity) because you don't have the necessary resources to make it happen(large swaths of memory and > 50ms response times).
I've done it and it's a pain in the ass. Why? Because the GC is _non-deterministic_. On many languages it can fire up at any time, freeze your threads at any time, and take however long it wants to. "But it usually doesn't take that long!" And? AND? The problem is, you've immediately added a blackbox thread attached to all your data. Enjoy having your program segfault in areas that are completely unrelated because the GC didn't run the finalizer till later. Enjoy having to fix a bug that doesn't have a repeatable effect. You're automatically writing multi-threaded code (and all the complexity that entails) even when you have a single-thread.
A GC adds an entire dimension to bug fixing and potential failure points. Every bug, multiplied by every possible interaction with the GC. This is something that systems programming usually cannot afford. You need to be able to prove your code won't crash.
And that's one more thing. Tons of systems code pre-allocates everything for that crash reason. So what problem is GC actually going to solve when everything is already carefully pre-allocated?
It really sounds like people who aren't used to unmanaged languages are simply projecting their fear of the unknown. I use both types on a daily basis, and to me they're just "the right tool for the right job." I would never try to shoehorn the wrong tool into an ill-fitted application just because I like that tool.