I'd guess that Swift also has something analogous to an array slice, which would be a (possibly) immutable borrow to a chunk of array data on the heap. This also happens to be a good use case for borrows.
From the other comment here it seems like Swift is pursuing an ownership model similar to Rusts, in which case, immutable borrows will become more important when you think about struct contents. You can only have a single owner, but you can specify many borrowers. This kind of thing is important when you have an array or vector of types, often you don't want those types to have a single owner but you want them to be populated or store a reference from somewhere else.
Anyway, don't dismiss the concept out of hand. Immutable borrows definitely have their uses, whether it's made explicit to you or not in Swift is another thing entirely.
Swift does have ArraySlice, but I don't get how borrows factor into that. Seems vaguely similar in concept, except ArraySlice exists to represent a subset of the original array, not just so you can pass arrays by reference. Since arrays are already passed by reference under the hood, that wouldn't really be useful.
I'm not dismissing the concept out of hand, so I'm not sure why you're warning me about that....
As far as large structures go, I'm thinking "large" like hundreds of fields. Any time I've seen people concerned about large structures, they misunderstand the value-typedness of things like String and Array and are worried about the contents of those things, which isn't really part of the size of the struct itself. But that's just what I've seen.