News for hackers. Stuff that matters.
Re:Swift 4.0 Released by sinhpham on Tuesday September 19, 2017 @11:00PM (#15290447)
They implemented what looks like the Rust ownership model: SE-0176 Enforce Exclusive Access to Memory (https://github.com/apple/swift-evolution/blob/master/proposa...), but I'm having a hard time understanding the proposal, can anyone shed some light on this?
Re:Swift 4.0 Released by slavapestov on Wednesday September 20, 2017 @12:09AM (#15290772)
You can think of inout parameters in Swift as something analogous to a mutable borrow in Rust. Until Swift 4 we allowed overlapping inout access, for example:

    var counter = 0
    func foo(x: inout Int) {
      x += 1
      print(counter)
    }
    foo(x: &counter)
Note how 'counter' is read by 'foo(x:)' during an inout access of the same value. This is now prohibited by Swift 4, using a combination of static and dynamic checks.

This fixes some undefined behavior and will also enable more aggressive compiler optimizations to be added in the future.

Re:Swift 4.0 Released by leshow on Wednesday September 20, 2017 @10:17AM (#15293957)
Isn't the mutable borrow ended after the end of the x += 1 line? leaving you free to read the contents of the ptr?

I don't know swift at all. but from their document on swapAt() it looks like they are trying to prevent two fn(&p, &p) where func fn(a: inout Type, b: inout Type)

Re:Swift 4.0 Released by hellofunk on Wednesday September 20, 2017 @11:04AM (#15294505)
> Isn't the mutable borrow ended after the end of the x += 1 line?

No, see Mike's comment here.

Re:Swift 4.0 Released by hellofunk on Wednesday September 20, 2017 @06:42AM (#15292285)
> Note how 'counter' is read by 'foo(x:)' during an inout access of the same value

It's not clear to me in your example why reading the value of counter after mutating it is bad; why is this now prohibited?

Re:Swift 4.0 Released by mikeash on Wednesday September 20, 2017 @10:48AM (#15294334)
Swift specifies inout parameters as copying the value that's passed in, giving the copy to the callee as a mutable value, and then writing back the value to the original storage after the function returns. & is not an "address of" operator.

Of course, it would be inefficient to do this all the time, so Swift will optimize copy-then-writeback to just passing a pointer to the original storage whenever it can. But this is an optimization operating under the "as if" rule: as long as it works as if it does a copy-then-writeback, the compiler can make it actually do whatever it wants.

If the example code were legal, then it would have to print `0`, because the writeback to `counter` doesn't happen until the end of the function. That means the compiler couldn't just pass in a pointer to `counter`, but would have to actually go through the copy-then-writeback procedure it's supposed to do, so you'd lose out on optimizations.

Instead, Swift makes it illegal. You can't access a value while this call is happening. That allows the language semantics to coexist with optimizations.

Re:Swift 4.0 Released by leshow on Wednesday September 20, 2017 @10:56AM (#15294409)
what does & do in Swift then? And what's the purpose of even having inout and & if you can't count on the code to actually be an address?
Re:Swift 4.0 Released by mikeash on Wednesday September 20, 2017 @11:02AM (#15294478)
& is just a sigil saying "I acknowledge that I am passing this as an inout parameter, and therefore the value may be modified by the function I am calling." Note that & is only legal on a function parameter. You cannot write, for example, `let b = &a`.

The purpose is to allow for out-parameters. A classic example would be the `+=` operator. (Swift operators are just normal functions with special call syntax.) It takes its first parameter as `inout` so that it can mutate the value.

Note that inout parameters work with expressions where it would be impossible to take the address. For example, you can use & on a computed property that has a setter. In that case it has to read the initial value, pass that to the function, then write back the new value, because it has no idea where the computed property actually stores the value, if anywhere.

Edit: because I'm obsessive and weird, I made a quick example of this computed property stuff:

http://swift.sandbox.bluemix.net/#/repl/59c284376cbea87f72c4...

Click the play triangle at the bottom to see the output.

Re:Swift 4.0 Released by leshow on Wednesday September 20, 2017 @02:14PM (#15296676)
> (Swift operators are just normal functions with special call syntax.)

Coming from Haskell and Rust it's nice to see this trend catching on.

Is Swift planning to introduce a distinction between borrows and mutable borrows to the user? From what you describe it seems like right now syntax-wise a borrow and a mutable borrow look the same, and the runtime makes some decision about it.

edit: Or I guess it could be the opposite. Since Swift passes by value always unless the runtime can optimize (right?), you could just not write & and inout and cross your fingers it gets optimized to a borrow rather than a copy?

Stuff like this makes me prefer the explicitness of Rust. It seems like here on the surface it's abstracted from you, but really you need to know the rules anyway or you could get into trouble.

Re:Swift 4.0 Released by pjmlp on Thursday September 21, 2017 @02:42AM (#15301045)
> Coming from Haskell and Rust it's nice to see this trend catching on.

It was already like that in older languages like Lisp, Smalltalk and CLU, just C++ made them in a different way.

Re:Swift 4.0 Released by mikeash on Wednesday September 20, 2017 @05:37PM (#15298543)
What's the use of a non-mutable borrow? Is it just for speed, to avoid copying a large structure? Large structures are rare in Swift and probably not important to optimize. (Value types like String and Array are actually just one reference under the hood, and that's all that gets "copied" when you pass those by value.)
Re:Swift 4.0 Released by dbaupp on Thursday September 21, 2017 @03:42AM (#15301275)
It also lets one avoid reference-counting traffic, which can be significant for values that contain multiple ARC/COW things (such as strings and arrays), and is semantically critical with move-only (or "unique ownership", per the ownership manifesto[1]) types.

[1]: https://github.com/apple/swift/blob/master/docs/OwnershipMan...

Re:Swift 4.0 Released by mikeash on Thursday September 21, 2017 @10:59AM (#15304081)
Unique ownership would be really nifty, and it makes sense that you'd definitely need pass by reference for it. Thanks for explaining!
Re:Swift 4.0 Released by leshow on Thursday September 21, 2017 @12:30PM (#15305096)
I don't know Swift, but non-mutable borrows are useful in a large number of languages. Are you sure a String is represented as a single reference? In Rust a String has a reference to the actual contents on the heap, a length, and a capacity. So that's a bit heavier than just a single reference, same thing with Vec. Rust's semantics are similar in that if you pass a Vec it will 'move' those 3 things, still, passing a string or vector slice (an immutable borrow) is faster, and moves less data, because it only copies a single reference.

I find it hard to believe that 'large structures are rare in Swift'. People don't make structs with multiple fields? You never want other structs to hold one or many of those? These are cases where non mutable borrows are useful. I understand that in Swift this is probably abstracted away from you and done by the runtime (or compiler) if it can, but that doesn't mean non-mutable borrows aren't useful; you just probably don't see them.

Of course, that's just a guess, I don't know Swift.

Re:Swift 4.0 Released by mikeash on Thursday September 21, 2017 @02:30PM (#15306188)
You're right, String has three fields. It's Array that's just a single reference. Is the overhead of passing three fields as a parameter so high that passing a reference is faster? Pointer chasing isn't free either, after all. I can certainly imagine scenarios where that sort of microoptimization pays off, but it seems like it would be rare.

As far as large structures go, I'm thinking "large" like hundreds of fields. Any time I've seen people concerned about large structures, they misunderstand the value-typedness of things like String and Array and are worried about the contents of those things, which isn't really part of the size of the struct itself. But that's just what I've seen.

Re:Swift 4.0 Released by leshow on Thursday September 21, 2017 @04:09PM (#15306911)
From the documentation of Array for Swift it has the same behaviour as Rust's Vec (it's growable, it allocates double the capacity after reaching max length). I'd find it pretty odd if it didn't share the same 3 word length as String. Also, how would it quickly know it's length if it didn't also store it's length in the same structure as the ptr to it's heap location? You'd have to chase 2 pointers just to get the length.

I'd guess that Swift also has something analogous to an array slice, which would be a (possibly) immutable borrow to a chunk of array data on the heap. This also happens to be a good use case for borrows.

From the other comment here it seems like Swift is pursuing an ownership model similar to Rusts, in which case, immutable borrows will become more important when you think about struct contents. You can only have a single owner, but you can specify many borrowers. This kind of thing is important when you have an array or vector of types, often you don't want those types to have a single owner but you want them to be populated or store a reference from somewhere else.

Anyway, don't dismiss the concept out of hand. Immutable borrows definitely have their uses, whether it's made explicit to you or not in Swift is another thing entirely.

Re:Swift 4.0 Released by mikeash on Thursday September 21, 2017 @05:07PM (#15307357)
The array capacity and length are stored inline before the contents. So you chase one pointer to get the capacity, length, or something stored in the array.

Swift does have ArraySlice, but I don't get how borrows factor into that. Seems vaguely similar in concept, except ArraySlice exists to represent a subset of the original array, not just so you can pass arrays by reference. Since arrays are already passed by reference under the hood, that wouldn't really be useful.

I'm not dismissing the concept out of hand, so I'm not sure why you're warning me about that....

Re:Swift 4.0 Released by hellofunk on Wednesday September 20, 2017 @10:59AM (#15294453)
Still new to Swift but I believe & is an explicit syntax necessary to make clear in the code that the function being called is mutating the argument, and thus the variable passed into that function could be mutated. It must be used wherever an argument is in/out. It makes mutation explicit both in the function declaration and also in the function call. Which is nice!

It's good for code readability but also prevents accidentally passing a variable to a function that could mutate it when you weren't expecting that, and vice-versa.

Re:Swift 4.0 Released by favorited on Wednesday September 20, 2017 @04:18PM (#15297834)
Values are not guaranteed to even have an address, IIRC.
Re:Swift 4.0 Released by hellofunk on Wednesday September 20, 2017 @11:04AM (#15294496)
Thanks Mike, very clear.
Re:Swift 4.0 Released by Falell on Wednesday September 20, 2017 @12:30AM (#15290874)
Note: I like to read about Rust but don't work with it seriously, and don't follow Swift at all. Corrections welcome. That said, these seemed like the key passages:

"Swift has always considered read/write and write/write races on the same variable to be undefined behavior. It is the programmer's responsibility to avoid such races in their code by using appropriate thread-safe programming techniques."

"The assumptions we want to make about value types depend on having unique access to the variable holding the value; there's no way to make a similar assumption about reference types without knowing that we have a unique reference to the object, which would radically change the programming model of classes and make them unacceptable for the concurrent patterns described above."

Sounds like a system in the vein of rust but more limited, with more runtime checks and no lifetime parameters, falling back to "programmer's responsibility" when things get hard. The last paragraph makes it sound like one of the motivations is in enabling specific categories of optimizations, as opposed to eliminating races at the language level.

One of my biggest questions as a reader is how a language like C handles these cases that Swift can't handle without these guarantees. Is this a move to get faster-than-C performance? Does C do these optimizations unsafely? Is there some other characteristic of Swift that makes this harder than C? Closures get a lot of focus in the article...

Re:Swift 4.0 Released by comex on Wednesday September 20, 2017 @03:14AM (#15291444)
The other responses to your comment are correct: C generally can't do those optimizations, unless you manually write `restrict`, and this can hinder optimization. But to complete the picture -

> Is there some other characteristic of Swift that makes this harder than C?

Yes:

1. Swift doesn't have pointers.

Instead, you have a lot of copying of value types, and the compiler has to do its best to elide those copies where it can. For instance, at one point the document mentions:

> For example, the Array type has an optimization in its subscript operator which allows callers to directly access the storage of array elements.

In C, C++, or Rust, you can "directly access the storage" without relying on any optimizations: just write &array[i] and you get a pointer to it. The downsides are (a) more complicated semantics and (b) the problem of what happens if array is deallocated/resized while you have a pointer to it. In C and C++, this results in memory unsafety; in Rust, the borrow checker statically rules it out at the cost of somewhat cumbersome restrictions on code.

2. Swift guarantees memory safety; C and C++ don't.

This goes beyond pointers. For instance, some of the examples in the document talk about potentially unsafe behavior if a collection is mutated while it's being iterated over. In Swift, the implementation has to watch out for this case and behave correctly in spite of it. In C++, if you, say, append to a std::vector while holding an iterator to it, further use of the iterator is specified as undefined behavior; the implementation can just assume you won't do that, and woe to you if you do. (In Rust, see above about the borrow checker. Iterator invalidation is in fact one of the most common examples Rust evangelists use to demonstrate that C++ is unsafe, even when using 'modern C++' style.)

Re:Swift 4.0 Released by pjmlp on Wednesday September 20, 2017 @04:54AM (#15291816)
> 1. Swift doesn't have pointers.

Sure it does:

    func modify(_ x:UnsafeMutablePointer<Int>) {
        x.pointee = 12;
    }
     
    func main()
    {
        var x = 23;
        print("Before \(x)\n");
        modify(&x);
        print("After \(x)\n");
    }
Re:Swift 4.0 Released by coldtea on Wednesday September 20, 2017 @07:14AM (#15292458)
Re:Swift 4.0 Released by hellofunk on Wednesday September 20, 2017 @06:32AM (#15292235)
> Swift guarantees memory safety

But memory leaks are quite easy to do with reference counting which is why Swift has some more complex syntax to prevent strong references. But it can take skill to understand when to use those techniques; the compiler doesn't always find these problems, thus there really isn't the guarantee you mentioned.

Re:Swift 4.0 Released by coldtea on Wednesday September 20, 2017 @07:13AM (#15292453)
Memory leaks are not the same as memory safety problems.
Re:Swift 4.0 Released by blub on Wednesday September 20, 2017 @07:25AM (#15292520)
Those Rust evangelists are only partially correct. It's the STL that's unsafe, not the language itself.

Iterators could be implemented in C++ in a safer way with some performance loss, but it doesn't seem to be a priority for anyone except the safercpp guy that posts here every now and then. STLs can enable iterator validation in a special debug mode.

Re:Swift 4.0 Released by lucian1900 on Wednesday September 20, 2017 @09:06AM (#15293285)
That is incorrect.

The language includes memory unsafe constructs without marking them in any way, since it must be compatible with C.

Re:Swift 4.0 Released by blub on Wednesday September 20, 2017 @11:38AM (#15294947)
We're discussing iterators here, and the STL iterators implemented as class templates can't even be compatible with C.

To clarify: safe containers, iterators and algorithms can be designed, but they don't seem to be a priority of the C++ community. Personally I'm quite scared of accidentally passing the wrong iterator to some function, but OTOH I can't recall it ever happening. I don't use the debug STL either, haven't needed it.

The examples that pcwalton keeps bringing up seem artificial to me. It's true that you can't have perfect safety in C++, but with some effort and custom libraries, many errors can be caught at compile or run-time. The advantage of Rust is that it's safe by default, not necessarily that there's a major safety difference between quality C++ and quality Rust.

Re:Swift 4.0 Released by comex on Wednesday September 20, 2017 @09:59PM (#15300013)
You’ve probably heard this before, but the security angle is important. “I haven’t had these problems in my code” really means “I haven’t triggered these problems in my code”… that is, unless you’ve had a security code audit done. Testing isn’t enough: even well-tested codebases can and do have vulnerabilities. In practice, they’re usually triggered by input that’s so nonsensical or insane from a semantic perspective, not only would it never happen in practice in ‘legitimate’ use, the code author doesn’t even think to test it. For a simple example, if some binary data has a count field that’s usually 1 or 2 or 10, what happens if someone passes 0x40000000 or -1? As a security researcher myself, I think it‘s actually easier to audit code with less knowledge of how the design is supposed to work, up to a point, because it leaves my mind more open. Rather than making assumptions about how different pieces are supposed to fit together, I have to look it up, and as part of looking it up I might find that the author’s assumptions were subtly wrong… For this reason, it’s really hard to audit your own code, at least in my experience. I mean, you can definitely keep reviewing it, building more and more assurance that it’s correct, but if your codebase is large enough, there may well be ‘that one thing’ you just never thought of.

I’m not actually sure how frequent iterator invalidation is as a source of vulnerabilities; I don’t think I’ve ever found one of that type myself. However, use-after-frees in general (of which iterator invalidation is a special case) are very common, usually with raw pointers. In theory you can prevent many use-after-frees by eschewing raw pointers altogether in favor of shared_ptr, but nobody actually does that – that’s important, because there’s a big difference between something being theoretically possible in a language and it being done in practice. (After all, modern C++ recommendations generally prefer unique_ptr or nothing, not shared_ptr!). And even if you do that, you can’t make the `this` pointer anything but raw, and same for the implicit raw pointer behind accesses to captured-by-reference variables in lambdas.

You can definitely greatly reduce the prevalence of vulnerabilities with both best practices for memory handling and just general code quality (that helps a lot). But if you can actually do that well enough - at scale - to get to no “major safety difference”, well, I haven’t seen the evidence for it, in the form of large frequently-targeted codebases with ‘zero memory safety bugs’ records. Maybe it’s just that C++’s backwards compatibility encourages people to build on old codebases rather than start new ones. Maybe. It’s certainly part of the story. But for now, I’m pretty sure it’s not the whole story.

Re:Swift 4.0 Released by blub on Thursday September 21, 2017 @08:21AM (#15302581)
C and C++ don't have a culture of safety, they have one of performance.

C++ code could be written significantly safer with a performance loss, e.g: index checking at run-time, iterator validity checking, exclusive smart ptr usage with null checking, etc. That, together with code reviews, static & dynamic analysis should IMO lead to comparable safety. That's what I'd do.

However, there doesn't seem to be a rush in that direction. My guess is that there won't be a rush to switch to Rust either.

Is the security angle that important that it's handled through education and better tooling? Or only important enough to do some code audits and pen testing?

Re:Swift 4.0 Released by jcelerier on Wednesday September 20, 2017 @12:06PM (#15295318)
> STLs can enable iterator validation in a special debug mode.

And they do actually, in the MSVC debug mode and with libstdc++'s -D_GLIBCXX_DEBUG. But nobody ever wants to use them.

Re:Swift 4.0 Released by smitherfield on Wednesday September 20, 2017 @01:59AM (#15291161)
C compilers must assume that a function pointer (that is, a function passed as an argument, or that is a property of an object) may write to any global variable.

C compilers must also assume that any two pointers to the same type may alias (refer to the same object). The programmer can assert to the compiler that a pointer does not alias any others used in the same scope by declaring it with the `restrict` keyword.

For most functions this won't have much effect on the generated code. Writing equivalent functions to the ones in the swift-evolution doc in C, both with and without `restrict` everywhere possible, it looks like `restrict` only has an effect on the generated code for `increaseByGlobal`: https://godbolt.org/g/W8s3BA

Re:Swift 4.0 Released by slavapestov on Wednesday September 20, 2017 @12:32AM (#15290880)
> One of my biggest questions as a reader is how a language like C handles these cases that Swift can't handle without these guarantees.

C doesn't address these issues at all as far as I know.

Re:Swift 4.0 Released by pjmlp on Wednesday September 20, 2017 @04:47AM (#15291783)
This is the first step into adopting more Rust like memory safety, but not at the expense of productivity.

Basically Swift will keep using reference counting as its GC algorithm, but for high performance situations it will be possible to have a bit more of fine grained control over ownership.

However they want to avoid any design that might result in "fighting with borrow checker" feeling.

Some info from WWDC 2017,

https://developer.apple.com/videos/play/wwdc2017/402/

There is also a transcript.

Re:Swift 4.0 Released by FlyingSnake on Wednesday September 20, 2017 @05:11AM (#15291869)
Re:Swift 4.0 Released by cbcoutinho on Wednesday September 20, 2017 @04:51AM (#15291802)
Here's the 'Ownership Manifesto' that tries to clarify some of the differences between the ownership models of Rust and Swift [1]. The main point raised in that document is how 'shared values' are being implemented in Swift in a less strict way compared to Rust.

[1] https://github.com/apple/swift/blob/master/docs/OwnershipMan...