As far as large structures go, I'm thinking "large" like hundreds of fields. Any time I've seen people concerned about large structures, they misunderstand the value-typedness of things like String and Array and are worried about the contents of those things, which isn't really part of the size of the struct itself. But that's just what I've seen.
I'd guess that Swift also has something analogous to an array slice, which would be a (possibly) immutable borrow to a chunk of array data on the heap. This also happens to be a good use case for borrows.
From the other comment here it seems like Swift is pursuing an ownership model similar to Rusts, in which case, immutable borrows will become more important when you think about struct contents. You can only have a single owner, but you can specify many borrowers. This kind of thing is important when you have an array or vector of types, often you don't want those types to have a single owner but you want them to be populated or store a reference from somewhere else.
Anyway, don't dismiss the concept out of hand. Immutable borrows definitely have their uses, whether it's made explicit to you or not in Swift is another thing entirely.
Swift does have ArraySlice, but I don't get how borrows factor into that. Seems vaguely similar in concept, except ArraySlice exists to represent a subset of the original array, not just so you can pass arrays by reference. Since arrays are already passed by reference under the hood, that wouldn't really be useful.
I'm not dismissing the concept out of hand, so I'm not sure why you're warning me about that....
I find it hard to believe that 'large structures are rare in Swift'. People don't make structs with multiple fields? You never want other structs to hold one or many of those? These are cases where non mutable borrows are useful. I understand that in Swift this is probably abstracted away from you and done by the runtime (or compiler) if it can, but that doesn't mean non-mutable borrows aren't useful; you just probably don't see them.
Of course, that's just a guess, I don't know Swift.