I’ve always thought Pin was difficult to understand because it’s not explained in a clear way in the official docs. In particular, lots of documentation claims things like “Pin ensures that an object is never moved”, which isn’t true!
It’s only true if the object is not Unpin, but most normal objects are Unpin, so Pin usually does nothing. It took me a very long time to finally understand this. The set of types T for which Pin<T> does anything at all is very niche and weird and IMO this isn’t sufficiently highlighted by the documentation.
I think this is good feedback and it would be good for the docs to be clearer about this. Of course for the types that you're going to deal with pinned (futures and streams), they're a lot more likely to be those niche objects.
I also do think the documentation has improved a lot over the years. I was surprised when I checked it while drafting this that it seemed to focus on the right things pretty well; circa 2019 I remember it being a lot more focused on specifying the contract in a way that really belongs in something like the Rust reference and not the std API docs.
My take on why users find Pin difficult: by itself, it has no meaning! This is different from every other wrapper in the language (except maybe AssertUnwindSafe<T>, which basically no one uses for its original purpose). Given a Pin<&mut InnerType>, there's nothing about Pin in the language or standard library that tells you what you can and can't do with it. (Unless the InnerType declares that it's Unpin, which implies that you can do anything you can do with an ordinary pointer.)
Instead, it operates as more of a "bring your own meaning", where the provider of the InnerType further creates any number of (internally unsafe) methods and APIs to manipulate a pinned object soundly. The only purpose of Pin<P> itself is to provide a pointer with fewer "intrinsic capabilities" (e.g., swapping &muts, moving out of Boxes, etc.), so that the inner type can allow further capabilities on top of that.
I suspect it's this nebulousness of meaning that confuses people the most. It certainly took me a fair while to figure it out. All the ideas about structural vs. non-structural fields are just to facilitate popular access patterns like "this one field is just plain old data, but this other field contains an object that itself wants to be pinned".
Pin does have meaning: it means you cannot move the target of this pointer ever again (or invalidate its without running its destructor, which is what moving does that's the problem), unless the type of the target implements Unpin.
This giving up of certain rights gives other rights (such as to store self-referential values), which are the reason you give it up. This is how contracts between components just work: similarly, giving up the right to mutate through a reference allows you to alias the reference at the same time. I'm always reminded of this line from Lincoln, about a very different and much graver subject: "If we submit to law, Alex, even submit to losing freedoms - the freedom to oppress for instance - we may discover other freedoms previously unknown to us."
I do agree that the fact that you can't use those rights in safe code is an educational problem, because one can't easily demonstrate what you can do with a pinned reference except "call a poll method which the compiler has generated for you."
> Pin does have meaning: it means you cannot move the target of this pointer ever again (or invalidate its without running its destructor, which is what moving does that's the problem), unless the type of the target implements Unpin.
Sure you can: it's just that the target has to provide its own methods for it. It's perfectly valid (if pointless) to write
struct PracticallyUnpin(..., PhantomPinned);
impl PracticallyUnpin {
fn unpin_mut(self: Pin<&mut Self>) -> &mut Self {
// SAFETY: We're the ones writing the rules here
unsafe { Pin::into_inner_unchecked(self) }
}
...
// (no other unsafe methods or impls)
}
and then the caller can do whatever they want with that reference, e.g., moving the value. Unpin isn't a magic word: it's just a generic way for the target to indicate that pinned pointers can safely regain full capabilities.
Of course, putting a Pin around an object does further restrict what you can do with it generically in unsafe code. But I'd further count these under the umbrella of "intrinsic unsafe capabilities", which Pin removes, but which the target can later restore ad libitum. Compare the question of whether a fn replace_with(&mut T, impl FnOnce(T) -> T) (aborting on panic) is sound, which really comes down to the "unsafe capabilities" of a &mut reference.
This is not to say, of course, that the target need not be very circumspect about which capabilities it restores! It has a responsibility not to create an unsound interface that might result in UB under permitted usage. E.g., if we have a type that owns a generic non-Unpin future, then that type must follow the strictest Pin invariants w.r.t. that future. But otherwise, it's entirely up to the target type which capabilities it wants to restore under which circumstances.
I've been programming Rust professionally for several years now and being totally honest I don't really understand Pin that well. I get the theory, but don't really have an intuitive understanding of when I should use it.
My usage of pin essentially comes down to "I try something, and the compiler complains at me, so I pin stuff and then it compiles."
It's never been such a hurdle in the day to day coding I need to do that I've been forced to sit down and truly grok it.
When teaching, to make it clear that an "Unpin" item is unaffected by "Pin," I’d suggest analogies from real life where objects remain unaffected despite the use of something designed to hold them in place:
1. Velcro hooks do not stick to smooth surfaces: Pin -> Velcro, Unpin -> Smooth
2. Magnets do not affect non-magnetic materials: Pin -> Magnet, Unpin -> NonMagnetic/Glass/Brass
3. Glue does not adhere to non-stick surfaces: Pin -> Glue, Unpin -> NonStick
This way, it becomes clear that a "Velcro" fixes an item in place, and if an item is "Smooth," it is unaffected by the "Velcro" mechanism.
Given Rust’s ecosystem naming themes it would have been beautiful to rename the trait to something something magnet and non-magnetic :’)
But a smooth object can't be velcroed, nor can wood hold up a magnet.
Isn't it that `Unpin` means the object is always ready to be pinned? (I read the article last night and already forgot whether pinning requires a fixup step)
So a `T: Pin + !Unpin` is like a sheet of paper that can only be fixed by stapling it, but a `T: Pin + Unpin` is like a painting with a hook which can be mounted on a nail and unmounted without damaging the hook
> The term “value identity” is not defined anywhere in this post, nor can I find it elsewhere in Mojo’s documentation, so I’m not clear on how Modular claims that Mojo solves the problem that Pin is meant to solve
I don't claim to know the answer either, but it reminds me of a great talk from Dave Abrahams, who worked on the value semantics for Swift together with Chris Lattner (who started Mojo). The talk is "Value Semantics: Safety, Independence, Projection, & Future of Programming" [0]
It's clear that Mojo is in some sense inheriting Swift's notion of "value semantics," but Rust also has "value semantics" in the same sense. Rust just also has references as first class types, whereas Swift (and as far as I can tell, Mojo) only allows references as a parameter passing mode; Mojo expands on Swift's inout parameters by having an immutable reference passing mode as well.
Not being able to store references in objects does solve the problem of "self-referential structs" in that you just can't implement code like the code Rust compiles to, but that isn't at all what the quoted paragraph says about Mojo so I am quite lost as to what they mean.
My understanding of _value identity_ refers to the `StableDeref`/yoke approach to self-regerential structs. The value is constructed at a stable address (usually some heap allocation) and you always access it through some pointer. The address is the value's identity.
The pointer can move, but the value doesn't move.
The problem as I see it is that having a &mut reference to something lets you move it via mem::swap/replace (and maybe a few others?) -- but actually needing to do so is rare. It seems to me that if that weren't allowed, taking a &mut reference to a self-referential value would be perfectly safe.
Maybe there could have been a way to opt in to moving-via-reference for those rare cases when you need it. Or maybe this whole problem could have been avoided by making swap and replace unsafe? I'd love to see someone explore that design space.
This is true. When were working on the problem, the way Aaron Turon put it was that &mut is "too powerful." If &mut didn't give the power to move out of it, the whole design would be a lot simpler. I'll discuss this in my next post about this.
Rust has to be backward compatible and already decided you can move out of an &mut, but there are definitely cleaner designs that can be, unburdened by what has been.
This just doesn't scale and it would have been a non starter because it would break so much exiting code. Mem swap is just one of the ways to move via mutable references, there are many many others. Option::take is one that I use quite often and it would be super weird if that was unsafe.
Its cool, but it also means the development of the language is very very slow. Async is still half backed and super complex, and I say that as a guy who has written rust code 40h/week for the past 3 years.
This isn't why development is slow. In my opinion, if I were still employed to work on Rust improvements to async would have shipped a lot faster. My blogging about it in my free time is my effort, in light of my circumstances, to get the project back to shipping on async.
I fully agree. Sometimes it feels like so much effort is put into rust's async side that the rest of the language ends up taking a back seat and suffering for it.
I can imagine a Rust-like language where we have move-constructors (in TFA), and every generated Future subtype is opaque and also heap allocated for us.
I think the need for Pin could then disappear, because the user would have no way to destroy it, since it's opaque and elsewhere on the heap, and therefore no way to move it (because having move-constructors implies that moving is conceptually destroying then recreating things).
You don't need move constructors if every future is heap allocated. But then every call to an async function is a separate allocation, which is terrible for memory locality. Some kind of virtual stack would be much better than that (but then you need garbage collection if you want the stacks to be memory-optimized to be small by default).
You could use an actual stack. As I understand it this was not done for questionable reasons relating to borrows of thread-locals. You could also allocate a top-level async function and all of its transitive async callees all at once, if you force the user to put all this information in 1 translation unit. Or you could use a bump allocator specifically for futures used in a certain part of the program, if you're willing to give up using a global allocator for everything. So it seems like there are a lot of options.
Yeah. I can guess how disruptive it would be, but I really wish rust bit the bullet and added a Move trait to std, baked into the language at a similar level as Copy. Move defines a function which moves a value from one address in memory to another. Structs without impl Move cannot be moved.
Almost all types would #[derive(Move)], which implements a trivial move function that copies the bytes. But this opens the door to self-referential types, futures, and lots of other things that need more complex move behaviour. (Actually, it might make more sense to define two traits, mirroring the difference between Copy and Clone. One is a marker trait which tells the compiler that the bytes can just be moved. The other allows custom "move constructor" implementation.)
I want move because pin is so hard to understand. Its a complex idea wrapped in double- or sometimes triple negatives. fn<X: !Unpin>(...). Wat? I drop off at unsafe pin-projecting. When is that safe? When is it not? Blah I'm out.
Moving from rust-without-move to rust-with-move would be inconvenient, because basically every struct anyone has written to date with rust needs #[derive(Move)] to be added. Including in std. And all types in existing editions that aren't pinned would need the compiler to infer a Move trait implementation. This should be possible to do mechanically. It would just be a lot of work.
Async rust is horrible. Especially compared to futures / promises in almost any other language. At some point, someone will make a new rust-like systems language which has an improved version of rust's memory safety model, a Move trait, and better futures. I'd personally also love comptime instead of rust's macro system.
I love rust. I love all the work the team has put into it over the years. But the language I’m really looking forward to is the language that comes after rust. Same idea, but something that has learned from rust’s mistakes. And it’s increasingly becoming clear what that better rust-like language might potentially look like. I can't wait.
I very much disagree that Pin is as hard as everyone makes it out to be. Using the pin! macros, the pin-project crate, and enough as_mut() to get it to compile and it's not hard at all to get a future impl working. It would be good to get this native (which is what boats wants) so it's easier to discover but it's not at all hard by any means
I think a lot of people think pin is confusing but don't actually try to learn it. When I've sat with people and helped them they understand pretty quickly what pin solves and how it works.
I very strongly think move constructors would be even more complex than pin.
> Async rust is horrible. Especially compared to futures / promises in almost any other language.
Having written a bunch of async code over 20 years (first exposure was twisted in the early 2000s and all sorts of stuff since - including manual function/stack wrangling on my own reactor), async in rust is like many other things in rust: It forces you to deal with the problems up-front rather than 6-12 months later once something is in prod. It helps to stop and understand why the compiler is yelling at you - for me anyway once I do grok that I'm glad I didn't have a repeat of $BUG_THAT_TOOK_DOWN_PROD_FOR_A_WEEK - that was a bad week.
Readit News
It’s only true if the object is not Unpin, but most normal objects are Unpin, so Pin usually does nothing. It took me a very long time to finally understand this. The set of types T for which Pin<T> does anything at all is very niche and weird and IMO this isn’t sufficiently highlighted by the documentation.
I also do think the documentation has improved a lot over the years. I was surprised when I checked it while drafting this that it seemed to focus on the right things pretty well; circa 2019 I remember it being a lot more focused on specifying the contract in a way that really belongs in something like the Rust reference and not the std API docs.
Instead, it operates as more of a "bring your own meaning", where the provider of the InnerType further creates any number of (internally unsafe) methods and APIs to manipulate a pinned object soundly. The only purpose of Pin<P> itself is to provide a pointer with fewer "intrinsic capabilities" (e.g., swapping &muts, moving out of Boxes, etc.), so that the inner type can allow further capabilities on top of that.
I suspect it's this nebulousness of meaning that confuses people the most. It certainly took me a fair while to figure it out. All the ideas about structural vs. non-structural fields are just to facilitate popular access patterns like "this one field is just plain old data, but this other field contains an object that itself wants to be pinned".
This giving up of certain rights gives other rights (such as to store self-referential values), which are the reason you give it up. This is how contracts between components just work: similarly, giving up the right to mutate through a reference allows you to alias the reference at the same time. I'm always reminded of this line from Lincoln, about a very different and much graver subject: "If we submit to law, Alex, even submit to losing freedoms - the freedom to oppress for instance - we may discover other freedoms previously unknown to us."
I do agree that the fact that you can't use those rights in safe code is an educational problem, because one can't easily demonstrate what you can do with a pinned reference except "call a poll method which the compiler has generated for you."
Sure you can: it's just that the target has to provide its own methods for it. It's perfectly valid (if pointless) to write
and then the caller can do whatever they want with that reference, e.g., moving the value. Unpin isn't a magic word: it's just a generic way for the target to indicate that pinned pointers can safely regain full capabilities.Of course, putting a Pin around an object does further restrict what you can do with it generically in unsafe code. But I'd further count these under the umbrella of "intrinsic unsafe capabilities", which Pin removes, but which the target can later restore ad libitum. Compare the question of whether a fn replace_with(&mut T, impl FnOnce(T) -> T) (aborting on panic) is sound, which really comes down to the "unsafe capabilities" of a &mut reference.
This is not to say, of course, that the target need not be very circumspect about which capabilities it restores! It has a responsibility not to create an unsound interface that might result in UB under permitted usage. E.g., if we have a type that owns a generic non-Unpin future, then that type must follow the strictest Pin invariants w.r.t. that future. But otherwise, it's entirely up to the target type which capabilities it wants to restore under which circumstances.
My usage of pin essentially comes down to "I try something, and the compiler complains at me, so I pin stuff and then it compiles."
It's never been such a hurdle in the day to day coding I need to do that I've been forced to sit down and truly grok it.
Whereas in C++ I was regularly treading in the water of "stuff I don't know, but must use" and getting eaten by crocodiles
1. Velcro hooks do not stick to smooth surfaces: Pin -> Velcro, Unpin -> Smooth
2. Magnets do not affect non-magnetic materials: Pin -> Magnet, Unpin -> NonMagnetic/Glass/Brass
3. Glue does not adhere to non-stick surfaces: Pin -> Glue, Unpin -> NonStick
This way, it becomes clear that a "Velcro" fixes an item in place, and if an item is "Smooth," it is unaffected by the "Velcro" mechanism.
Given Rust’s ecosystem naming themes it would have been beautiful to rename the trait to something something magnet and non-magnetic :’)
Isn't it that `Unpin` means the object is always ready to be pinned? (I read the article last night and already forgot whether pinning requires a fixup step)
So a `T: Pin + !Unpin` is like a sheet of paper that can only be fixed by stapling it, but a `T: Pin + Unpin` is like a painting with a hook which can be mounted on a nail and unmounted without damaging the hook
- the double negative of „!Unpin”
- Rust trait names are not adjectives
Pin + Detach(-able) would be a less confusing name.
That being said, you can velcro smooth objects all you want, but you'll still separate (Move) them easily :) or put a magnet to glass/brass objects.
PS mind you that Pin is not a trait, only Unpin is, so Pin<T: Unpin> or Pin<T: !Unpin> is a more accurate way or writing what you described :)
I don't claim to know the answer either, but it reminds me of a great talk from Dave Abrahams, who worked on the value semantics for Swift together with Chris Lattner (who started Mojo). The talk is "Value Semantics: Safety, Independence, Projection, & Future of Programming" [0]
[0] https://www.youtube.com/watch?v=QthAU-t3PQ4
Not being able to store references in objects does solve the problem of "self-referential structs" in that you just can't implement code like the code Rust compiles to, but that isn't at all what the quoted paragraph says about Mojo so I am quite lost as to what they mean.
Maybe there could have been a way to opt in to moving-via-reference for those rare cases when you need it. Or maybe this whole problem could have been avoided by making swap and replace unsafe? I'd love to see someone explore that design space.
Rust has to be backward compatible and already decided you can move out of an &mut, but there are definitely cleaner designs that can be, unburdened by what has been.
It feels like there's very few communities that people going so in depth publically about the nitty gritty of their language, and it's so cool to see.
I think the need for Pin could then disappear, because the user would have no way to destroy it, since it's opaque and elsewhere on the heap, and therefore no way to move it (because having move-constructors implies that moving is conceptually destroying then recreating things).
This has a very nice effect of allowing merging and inlining of Futures before they're executed.
It's similar to how Rust does immutability — there's no immutable memory, only immutable references.
Almost all types would #[derive(Move)], which implements a trivial move function that copies the bytes. But this opens the door to self-referential types, futures, and lots of other things that need more complex move behaviour. (Actually, it might make more sense to define two traits, mirroring the difference between Copy and Clone. One is a marker trait which tells the compiler that the bytes can just be moved. The other allows custom "move constructor" implementation.)
I want move because pin is so hard to understand. Its a complex idea wrapped in double- or sometimes triple negatives. fn<X: !Unpin>(...). Wat? I drop off at unsafe pin-projecting. When is that safe? When is it not? Blah I'm out.
Moving from rust-without-move to rust-with-move would be inconvenient, because basically every struct anyone has written to date with rust needs #[derive(Move)] to be added. Including in std. And all types in existing editions that aren't pinned would need the compiler to infer a Move trait implementation. This should be possible to do mechanically. It would just be a lot of work.
Async rust is horrible. Especially compared to futures / promises in almost any other language. At some point, someone will make a new rust-like systems language which has an improved version of rust's memory safety model, a Move trait, and better futures. I'd personally also love comptime instead of rust's macro system.
I love rust. I love all the work the team has put into it over the years. But the language I’m really looking forward to is the language that comes after rust. Same idea, but something that has learned from rust’s mistakes. And it’s increasingly becoming clear what that better rust-like language might potentially look like. I can't wait.
I think a lot of people think pin is confusing but don't actually try to learn it. When I've sat with people and helped them they understand pretty quickly what pin solves and how it works.
I very strongly think move constructors would be even more complex than pin.
Having written a bunch of async code over 20 years (first exposure was twisted in the early 2000s and all sorts of stuff since - including manual function/stack wrangling on my own reactor), async in rust is like many other things in rust: It forces you to deal with the problems up-front rather than 6-12 months later once something is in prod. It helps to stop and understand why the compiler is yelling at you - for me anyway once I do grok that I'm glad I didn't have a repeat of $BUG_THAT_TOOK_DOWN_PROD_FOR_A_WEEK - that was a bad week.