Why? In C I'd understand. But cargo and the swift package manager work great.
By all means, rewrite little libraries instead of pulling in big ones. But if you're literally copy+pasting code between projects, it doesn't take much work to pull that code out into a shared library.
I don't know what the solution is, but these days I'm a lot more likely to simply copy code over to a new project rather than try to build general purpose libraries.
I feel like that's part of the mess Rust/Swift are getting themselves tangled up in, everything depends on everything which turns evolution into more and more of an uphill struggle.