News for hackers. Stuff that matters.
Re:Unsoundness in Pin by kibwen on Friday November 29, 2019 @10:53AM (#21665039)
For those who haven't heard of `Pin` before (which is reasonable even for Rust users, for whom this is a rather obscure corner of the standard library): it's essentially a trait that indicates that a value cannot be moved from its current location in memory.

Here's an analogy: Rust already differs from most languages in that the behavior that you get "by default" when defining a new type is very limited. For example, if I define a type `struct Foo`, then by default this type can't be copied around: `let x = Foo; let y = x;` is a simple operation that in most languages would involve copying memory, but in Rust involves a move instead. You can think of a move as a copy where the original value is no longer accessible (whether or not any bytes in memory actually get copied as a result of a move is an implementation detail, and is (hopefully) often optimized away). In order to make our type copyable, we would add an implementation of the `Copy` trait.

The observation is that moving is a more fundamental operation than copying, therefore, it is easier for types to opt in to copying than to opt out.

When Rust was released in 2015 this was a relatively radical concept for a language targeting a mainstream audience (we're essentially describing Rust's entire concept of ownership here, after all). But it turns out that it's possible that Rust may not have been radical enough! Consider: what if you want a type that not only can't be copied, but can't be moved as well?

Before Rust 1.0 it wasn't clear that such a concept of "unmoveability" would be generally useful. There were certainly cases, such as self-referential structs, that could have benefited from such a concept, however making moveability opt-in rather than opt-out would have required all types that do want to be moveable to explicitly announce that fact (via something like `#[derive(Move)]`), which is an annotation burden on all other code that must be considered.

It wasn't until the async/await work that another, more critical need for unmoveability was found: if the generators interally produced by `async` are capable of moving, then that means that generators are incapable of containing references, which means that async/await would become drastically less useful; there would be an entire fundamental part of the language that simply couldn't be used with it. Thus `Pin` was born in order to denote things that cannot be moved (and whose design is itself a very, very long discussion).

For this reason I'm somewhat amused by one of the comments in the OP asking something like "if `Pin` had remained unstable for an additional year, would this particular instance of unsoundness have been caught?" Because, conversely, we could ask whether Rust itself could have remained unstable for another five years and found a design that would have obviated `Pin` entirely by making moveability opt-in. However, of course, such things are easy to ask in hindsight, and stability is a prerequisite for having a broad base of adoption. Finding a balance between immediate stability and eventual perfection is the holy grail of industrial language design.

(Regarding the bug in the OP itself, I think it's unfortunate but I'm not especially worried by it at this juncture. The fact that Ralf Jung's team is looking into it gives me confidence that formal methods will eventually be applied here to more thoroughly explore the soundness of `Pin` in general (Ralf being one of the people who shaped `Pin` originally), and in the meantime I wouldn't be opposed to a band-aid fix, given that working extensively with `Pin` in the way shown in the OP is rare for normal users.)

Re:Unsoundness in Pin by jeltz on Friday November 29, 2019 @11:08AM (#21665134)
> Before Rust 1.0 it wasn't clear that such a concept of "unmoveability" would be generally useful. [...]

Hm, I ran into quite a lot of cases where I wanted Pin or something similar when porting a C program to Rust, and if I recall correctly also the creator of vulkano also ran into plenty of cases when writing it, so I am a bit surprised people did not realize this. My case was a program which in its C version was based around an epoll event loop, so maybe that is a bit similar to the async/generators issue.

Re:Unsoundness in Pin by kibwen on Friday November 29, 2019 @11:15AM (#21665163)
I could have perhaps written "would be generally useful enough to justify the additional annotation burden" (alluded to later in the paragraph). Also, in what year were you discovering these use cases? The discussions in question would have had to have happened in 2014 or earlier; for people who began using Rust post-1.0, it may very well be that these use cases were encountered because the language was stable by then, and that people might not have bothered enough with an unstable language to bother pushing the language to its limits like this. Though I guess we can take solace in the idea that the language that comes after Rust someday might benefit from these case studies. :)
Re:Unsoundness in Pin by jeltz on Friday November 29, 2019 @11:17AM (#21665174)
You are very correct in your guess that it was post-1.0, it is like you can read my mind. :) I would indeed not have bothered to port my production code to a non-stable language. But once I started doing that I ran into these issues quite quickly.
Re:Unsoundness in Pin by epage on Friday November 29, 2019 @11:05AM (#21665114)
> But it turns out that it's possible that Rust may not have been radical enough! Consider: what if you want a type that not only can't be copied, but can't be moved as well?

When talking about Pin, I think it is important to go back to a more fundamental level.

There are types that are not movable without extra processing, like a `struct` that has fields that reference other fields. When moving, those references become invalid.

Rust chose by allowing types to become unmovable so the `struct` can't go into an invalid state.

While C++ doesn't have the same safety guarantees, they did need to solve this problem. C++ chose to solve this with move constructors. I know move constructors will have their own challenges in Rust but I hope we do get them one day.

Re:Unsoundness in Pin by zozbot234 on Friday November 29, 2019 @11:14AM (#21665161)
There is an unofficial 'transfer' crate that builds on Pin<> and on a further unofficial "stackpin" feature (pinning objects on the stack, which is currently not provided for in standard Rust) to provide an equivalent to C++ move constructors.
Re:Unsoundness in Pin by creatonez on Friday November 29, 2019 @01:00PM (#21665813)
I don't get why we don't just use the same magic that `?Sized` uses, to make it opt-out in terms of mental overhead for the developer, but opt-in in terms of what's going on under the hood:

    #[derive(!Move)]
Re:Unsoundness in Pin by jeltz on Friday November 29, 2019 @01:16PM (#21665902)
Maybe this wont work for Move speficially, but I like this a solution for the general problem of skipping default traits.
Re:Unsoundness in Pin by withoutboats on Sunday December 01, 2019 @07:02PM (#21678712)
The syntax is different but Rust has this feature- its called auto traits, and its how we handle traits related to threadsafety. Its not a solution for what Pin solves because the whole notion that a type might be moveable or not moveable is actually a misstatement of the problem that Pin is solving.
Re:Unsoundness in Pin by cesarb on Friday November 29, 2019 @01:33PM (#21665978)
> [...] would have required all types that do want to be moveable to explicitly announce that fact (via something like `#[derive(Move)]`) [...]

Isn't `Pin` even crazier, in that the same type can be moveable (when used outside of Pin) or not moveable (when used inside Pin)? A simple derive wouldn't be enough to capture that.

Re:Unsoundness in Pin by kibwen on Friday November 29, 2019 @02:34PM (#21666298)
I wouldn't say it's crazy, or even unprecedented. Consider the reference-counted `Rc` smart pointer type, which allows a given type to act as though it were cloneable (when used inside of `Rc`) or not cloneable (when used without `Rc`).

The hypothetical opt-in moveability proposal is difficult to compare directly to `Pin` because they take such different approaches to the problem, but they would both address the problem (at least, as far as I have considered it).

Re:Unsoundness in Pin by jimmaswell on Friday November 29, 2019 @11:59AM (#21665432)
> Before Rust 1.0 it wasn't clear that such a concept of "unmoveability" would be generally useful

I'd like to ask, why is moving generally useful? Why would I want to move x to y and invalidate x?

Re:Unsoundness in Pin by steveklabnik on Friday November 29, 2019 @12:05PM (#21665464)
We can go back to the reason that Pin even exists in the first place. With Rust, you create a computation, and it implements the Future trait. When you want it to execute, you give it to an executor. This often moves the data structure that represents the computation to the heap, so that it has a stable address, and can continue running after the current stack frame is over. So, before the future starts executing, it can be moved around in memory (and probably will at least one time), but after it starts executing, it must never move again.

Does that make sense?

Re:Unsoundness in Pin by jimmaswell on Friday November 29, 2019 @12:53PM (#21665776)
So the main use is convenience to seamlessly make stack variables permanent? Intuitively that doesn't sound worth the language design troubles compared to making the programmer declare these things on the heap manually. I'm not that familiar with Rust though.
Re:Unsoundness in Pin by steveklabnik on Friday November 29, 2019 @01:02PM (#21665826)
That’s one use case, it’s not the only one. And it’s not always “put it on the heap”, so you can’t solve the issue that way.
Re:Unsoundness in Pin by kibwen on Friday November 29, 2019 @02:37PM (#21666319)
Ideally it should be possible to avoid the heap entirely; futures in Rust should be capable of operating even in contexts that don't come with a memory allocator, such as memory-constrained embedded devices.
Re:Unsoundness in Pin by dodobirdlord on Friday November 29, 2019 @03:05PM (#21666512)
Rust’s ownership model is used to provide memory safety. Entities have only one owner so that the compiler can reason about when they go out of scope and can be safely dropped. If x didn’t get invalidated by the move then both x and y would own the entity. You could then do something like pass y into a function that would drop y at the end of the function scope, and then cause a use-after-free by using x. With the ownership model the compiler can guarantee that use-after-free is always a syntax error. The ownership model is a fair bit more complicated than this, but move semantics are at the core of it.