C# is better than many alternatives, but Swift is simply a better language in every regard.
C#'s GC is a constant issue for games, and it makes using things like LINQ nearly impossible since it does so much allocation. Swift is designed around automatic reference counting instead. C#'s structs work weirdly and are hard to use.
Swift is so much more ergonomic. C# has added a few nice things lately, but it's much more verbose than Swift.
Swift is also more focused on performance, and is always AOT compiled.
Unity has had to come up with some insane things to get C# to work across platforms – they have their own .NET IL to C++ compiler (https://docs.unity3d.com/Manual/IL2CPP.html) that doesn't always work right. They also have a SECOND custom C# compiler for high-performance programming (https://docs.unity3d.com/Packages/[email protected]/manual...).
I'm also very excited about Swift 6's upcoming opt-in Rust-like lower-level memory management which should help a ton for performance-critical code. (https://github.com/apple/swift/blob/main/docs/OwnershipManif...)
They've recently announced that they're finally upgrade to .net core over the next couple years, so that should help out a lot. https://blog.unity.com/technology/unity-and-net-whats-next
There are much better languages for game scripting (LUA for example)
C# is a robust language with a lot of features like lambdas pattern matching etc. I also find that most of the people who know Swift are Apple developers, so I feel like there isn't a broad enough appeal especially for game devs who are going to be more Windows or Linux centric.