So tell me, how would the invalidation work if the API were written in a language like Rust?

You've just listed a bunch of GCed languages and told me it's due to the API not the language. GC lets you be sloppy with resource initialization/cleanup though, and having GC doesn't magically fix when the external objects you interact with go in and out of scope.

So what I'm arguing above is that GCed languages are actually just as hard as manual memory managed languages the moment you need to interact with the real world... be it UI elements, file handles, network sockets, or whatever. Memory isn't the only resource that needs management, and languages that have idiomatic resource management are actually good at this kind of thing.