Yep, I've only started using Swift a couple of months ago, but Xcode just isn't pleasant to use imo (though I'm probably spoiled by Jetbrains' IDEs), and the Swift extension for VS Code is clunky.
I'm still really sad Jetbrains decided to sunset AppCode :(
So the quality of tooling/IDEs is definitely a factor, I just don't see myself using Swift outside of the Apple ecosystem when there are so many other alternatives.
You don't use Xcode to write Swift, you use Xcode to write Apple apps which you happen to do in Swift. No matter the language, you need Xcode to write those apps.
Equalling Xcode and Cargo makes no sense. A similar situation to Xcode and Swift would be like Visual Studio and C# on Windows. Many developers use these tools, but they are not the language ecosystem, and the Cargo equivalents are entirely separate (Nuget, Cocoapods).
If you argue about nomenclature, yes, Xcode and cargo fit into entirely different categories.
However, they are still comparable you can't feasibly escape either to develop Swift or Rust apps the way most people develop them.
You could also argue that e.g. Rust is not cargo, but almost every Rust programmer will be using cargo. Sure I could use something else, but why would I? Of course the analogy is not perfect, because the "why would I?" is clear for Xcode: it's bad and macOS only :D
All I'm trying to say is: defaults matter. Most people will not be writing Swift in VS Code (on macOS).