I'm very much a fan of the idea that language features — and especially library features — should not have privileged access to the compiler.
Rust is generally pretty good at this, unlike (say) Go: most functionality is implemented as part of the standard library, and if I want to write my own `Vec` then (for the most part) I can. Some standard library code relies on compiler features that haven't been marked stable, which is occasionally frustrating, but the nightly compiler will let me use them if I really want to (most of the time I don't). Whereas in Go, I can't implement an equivalent to a goroutine. And even iterating over a container was "special" until generics came along.
This article was a really interesting look at where all that breaks down. There's obviously a trade-off between having to maintain all the plumbing as user-visible and therefore stable vs purely magic and able to be changed so long as you don't break the side effects. I think Rust manages to drive a fairly good compromise in allowing library implementations of core functionality while not needing to stabilise everything before releasing anything.
> I'm very much a fan of the idea that language features — and especially library features — should not have privileged access to the compiler.
At some point I realized I was in the opposite camp and nothing I have seen since that has really changed my view point.
Languages: compilers, libraries, toolkits, etc, aren't supposed to be some abstract collection of parts that can be theoretically hooked together in any possible way to achieve any possible result, they're for solving problems.
You can argue that these things are not opposites, and in theory that's true, but in practice, they seem to be! Go is a good example of making compromises that limit flexibility for the sake of developer/designer convenience.
An interesting example would be Lego, I'd argue that Go is closer to Lego design because it has a bunch of specific pieces that fit together but only in the way the designer intended.
I suspect someone taking the opposite approach from, say, Rust, would argue that some Go pieces don't actually fit together in the way that we think all lego pieces should.
My counter argument is that not all Lego pieces do actually fit together and, like, you can't cut a piece in half to make a new piece that just doesn't exist. You're limited to what comes out of the factory.
On the other hand, much to my childhood self's chagrin, even when opening a fresh set and not mixing it with other Legos, there are still quite a large number of other ways to put the pieces to together. After multiple decades my mother still sometimes tells an anecdote about how flabbergasted I was at a friend in kindergarten who opened a set I got him as a gift, ignored the instructions, and proceeded to build some bespoke edifice with virtually no resemblance to the picture on the box.
The sheer number of combinations exposed by the pieces that can number in the thousands for some sets that can be assembled either as the designer intended, completely differently, or even mixed and matched with other pieces from other sets to me feels far more like the parent comment description of exposing the underlying features, shipping a specific vision of them being assembled in a certain way, but allowing alternative visions to be constructed. What you're describing to me is more akin to building blocks; they're larger, more uniform, and only possible to put together in the ways that the designers intended. Stacking them up requires far less effort than putting together a Lego set, and you're not going to have trouble understanding how to stack some pieces so that they're relatively stable, but you're limited by the combinations of ways you can compose uniform cubes. You can't build an arch or bridge because gravity will get in the way, and you don't have the ability to make any other shapes out of the block material.
> At some point I realized I was in the opposite camp and nothing I have seen since that has really changed my view point.
I'm in this camp as well. The additional machinery required to make library features pluggable often adds a lot of complexity to the user-visible semantics of the language. And it's often not so easy to limit that complexity to only the situations where the user is trying to extend the language.
That's not always true. Sometimes the process of making seemingly core features user-replaceable will reveal simpler, more fundamental abstractions. But this article is a good example of the opposite.
One might push that analogy past its limit, and suggest that you probably wouldn't try to build a road-legal car out of Lego :).
More seriously, I'm not expecting that most people will want to use the underlying language features, nor indeed that those who do should actually do so commonly. They're there to provide for a clean separation between control logic and business logic. And that helps us to create cleaner abstractions so we can test our control logic independently of our business logic.
Most of the time I should be using the control logic we've already written, not writing more of it.
What you are missing is the whole point of writing better languages is to write better libraries.
It is not worth it to write better languages if just the "last mile" of final program is more productive --- making new languages is extraordinarily expensive, and this sort of single-shot productivity boost just isn't worth it.
If you can make better libraries --- code reuse and abstractions at hitherto unimaginable ways and then use those libraries to write yet more libraries, you get a tower of abstractions, each yielding every more productivity, like an N stage rocket.
This sort of ever-deepening library ecosystem is scarcely imaginable to most programmers. Or, they think they can imagine it, but everything looks like a leftpad waste of time to them.
Having a separation between the "pure language" and the library is a requirement if you want to have a language that can be used for low-level components, like kernels or bare-bones software.
I don't think this is possible in a language that needs a runtime, like Go.
I've been struggling with a related problem over at [1].
Feel free to read this, but it's nowhere near finished.
I'm trying to figure out how to do back references cleanly and safely.
The basic approach I'm taking is
- We can do just about everything useful with Rc, Weak, RefCell, borrow(), borrow_mut(), upgrade, and downgrade. But it's really wordy and there's a lot of run time overhead. Can we fix the ergonomics, for at least the single-owner case?
Probably. The general idea is to be able to write a field access to a weak link as
sometype.name
when what's happening under the hood is
sometype.upgrade().unwrap().borrow().name
- After fixing the ergonomics, can we fix the performance by hoisting some of the checking? Probably. It's possible to check at the drop of sometype whether anybody is using it, strongly or weakly. That allows removing some of the per-reference checking. With compiler support, we can do even more.
What I've discovered so far is that the way to write about this is to come up with real-word use cases, then work on the machinery. Otherwise you get lost in type theory.
The "Why" has to precede the "How" to get buy-in.
> The general idea is to be able to write a field access to a weak link as
sometype.name
> when what's happening under the hood is
sometype.upgrade().unwrap().borrow().name
You could easily implement this with no language-level changes as an auto-fixable compiler diagnostic. The compiler would error out when it sees the type-mismatched .name, but it would give you an easy way of changing it to its proper form. You just avoid making the .name form permanent syntactic sugar (which is way too opaque for a low-level language like Rust), it gets replaced in development.
.borrow() already has a hidden unwrap. There's try-borrow(), but the assumption for .borrow() is that it will always succeed.
What I'd like to do is move as much of the checking as possible to the drop() of the owning object, and possibly to compile time. If .borrow() calls are very local, it's not too hard to determine that the lifetimes of the borrowed objects don't overlap.
Upgrade is easy to check cheaply at run time for Rc-type cells. Upgrade only fails if the owning object has been dropped. At drop, if weak_count == 0, no dangling weak references outlive the object. If there are more strong references, drop would not be called. With that check, .upgrade() will never fail.
After all, when a programmer codes a potentially fatal .borrow(), they presumably have some reason to be confident the panic won't trigger.
For those trying to grok their stuff after reading the blog post, consider this.
The borrow checker vs type checker distinction is a hack, a hack that works by relegating a bunch of stuff to be "second class". Second class means that the stuff only occurs within functions, and never across function boundaries.
Proper type theories don't have this "within function, between function" distinction. Just as in the lambda calculus, you can slap a lambda around any term, in "platonic rust" you should be able to get any fragment and make it a reusable abstraction.
The author's here lens is async, which is a good point that since we need to be able to slice apart functions into smaller fragments with the boundaries at await, we need this abstraction ability. With today's Rust in contrast, the only way to do safe manual non-cheating awake would instead to be drasticly limit where one could "await" in practice, to never catch this interesting stuff in action.
In my thing I hadn't considered async at all, but was considering a kind of dual thing. Since these inconsievable types do in fact exist (in a Rust Done Right), and since we can also combine our little functions into a bigger function, then the inescable conclusion is that locations do not have a single fixed type, but have types that vary at different points in the control flow graph. (You can try model the control flow graph as a bunch of small functions and moves, but this runs afowl of non-movable stuff, including borrowed stuff, the ur-non-moveable stuff).
Finally, if we're again trying to make everything first class to have a language without cheating and frustration artificial limits on where abstraction boundaries go, we have to consider not just static locations changing type, but also pointers changing type. (We don't want to liberate some types of locations but not others.) That's where my thing comes in — references that have one type for the pointee at the beginning of the lifetime, and another type at the end.
This stuff might be mind blowing, but if should be seriously pressude. Having second class concepts in the language breeds epiccycles over time. It's how you get C++. Taking the time to make everything first class like this might be scary, but it yields a much more "stable design" that is much more likely to stand the test of time.
The post concludes by saying it's hopeless to get this stuff implemented because back compat, but I do think that that is true. (It might be hopeless for other reasons. It certainly felt hopeless in 2015.)
All this is about adding things to the language. That's backwards compatible. E.g. Drop doesn't need to be changed, because from every Drop instance a DropVer2 instance can be written instead. async v1 can also continue to exist, just by continuing to generate it's existing shitty unsafe code. And if someone wants something better, they can just use async v2 instead.
People get all freaked out about changing languages, but IMO the FUD is entirely due to sloppy imperative monkey brain. Languages are ideas, and ideas are immutable. The actual question is always, can we do "safe FFI" between two languages. Safe FFI between Rust Edition 20WX and 20YZ is so trivial that people forget to think about it that way. C and C++ is better since C "continues to exist", but of course the bar for "safe FFI" is so low when the language themselves are unsafe within themselves so that safety between them couldn't mean very much.
With harder edition breaks like this, the "safe FFI" mentality actually yields fruit.
I've considered rust for some performance-critical greenfield work where I would normally use C. Rust's syntax, idioms, and packaging are foreign to my team, so the only motivation to take that on is the safety of the borrow checker.
But as I investigate Rust, I learn of trivial use cases that cannot be safely represented [0] in Rust's syntax. TFA demonstrates even more provably-safe techniques that are impossible to express safely in Rust. So after all the difficulty of learning Rust, I might still have to choose between performance and safety?
My impression is that Rust 1.91.0 simply isn't a suitable C replacement for many real world use cases. But since it's already being used in production, I worry that backwards compatibility concerns will prevent these issues from being fixed properly, or at all.
Perhaps rust2 will get this right. Until then there's C.
I think they should just implement position-independent borrows. So instead of the borrow being an absolute pointer that gets broken if you move the self-borrowing struct, you can move it just fine.
Yes it would add like one extra add to every access, but you hardly ever need self-borrows so I think it's probably an acceptable cost in most cases.
struct A {
raw_data: Vec<u8>,
parsed_data: B<&pie raw_data>,
parsed_data2: B<&pie raw_data>
}
struct B<T> {
foo: &pie T [u8],
}
Ignoring that my made up notation doesn't make much sense, is the idea that B.foo would be an offset relative to its own adress?
So B.method(&self) might do addr(&self.foo) + self.foo, which is stable even if the parent struct A and its raw data field moves?
Then I wonder how to handle the case where the relative &pie reference itself moves. Maybe parsed_data is std::mem::replaced with parsed_data2 (or maybe one of them is an Option<B> and we Option.take() it somewhere else.)
This has been proposed at the time, but it doesn't work for the case where the borrow points to stable memory (e.g. a `&str` pointing to the contents of a `String` in the same struct). In general case a reference might point to either stable or unstable memory at runtime, so there's no way to make this always work (e.g. in async functions)
Some of the things that makes Rust hard to wrap you head around are the very things this post elucidates.
The principles exposed by the language, like lifetimes and mutability, are easy enough to understand. Yet despite understanding them, you get into epic battles with the borrow checker where the new shiny rules you've just learnt turn out to be little help in figuring out why the compiler accepts some code but rejects something similar.
You can't figure it out because these inconceivable types are not written down or explained anywhere. Instead if you are lucky you get examples of what works and what doesn't along with a hand wavy description of why, but usually it's just you versus the compiler error message. Since you don't have a concrete description of what is going to help you reason about what it likely to be accepted, you resort to experimentation to try and find what the compiler does accept. The experimental route takes ages.
If you survive enough bruising rounds of this, you will build up an empirical list of ways of working in Rust. If not, you give up on the language. To be fair, I don't think it takes a huge amount of time in the scheme or things, certainly less time than it takes to know your way around a languages standard library as that can take a month or two. However for those of us with a few languages under our belts, the time it does take comes as a shock. I'm used to it taking a few hours to become acquainted with the syntax and semantics or a new language, not finding myself still having battles with the compiler weeks later.
IMO, if the Rust doco did explain the language things the way this post does, it would make the learning it much easier.
The syntax in this post is hypothetical. In common usage you’d never encounter a need to even think about these complexities, let alone a desire to do the manual work discussed in this blog post.
Readit News
Rust is generally pretty good at this, unlike (say) Go: most functionality is implemented as part of the standard library, and if I want to write my own `Vec` then (for the most part) I can. Some standard library code relies on compiler features that haven't been marked stable, which is occasionally frustrating, but the nightly compiler will let me use them if I really want to (most of the time I don't). Whereas in Go, I can't implement an equivalent to a goroutine. And even iterating over a container was "special" until generics came along.
This article was a really interesting look at where all that breaks down. There's obviously a trade-off between having to maintain all the plumbing as user-visible and therefore stable vs purely magic and able to be changed so long as you don't break the side effects. I think Rust manages to drive a fairly good compromise in allowing library implementations of core functionality while not needing to stabilise everything before releasing anything.
At some point I realized I was in the opposite camp and nothing I have seen since that has really changed my view point.
Languages: compilers, libraries, toolkits, etc, aren't supposed to be some abstract collection of parts that can be theoretically hooked together in any possible way to achieve any possible result, they're for solving problems.
You can argue that these things are not opposites, and in theory that's true, but in practice, they seem to be! Go is a good example of making compromises that limit flexibility for the sake of developer/designer convenience.
An interesting example would be Lego, I'd argue that Go is closer to Lego design because it has a bunch of specific pieces that fit together but only in the way the designer intended.
I suspect someone taking the opposite approach from, say, Rust, would argue that some Go pieces don't actually fit together in the way that we think all lego pieces should.
My counter argument is that not all Lego pieces do actually fit together and, like, you can't cut a piece in half to make a new piece that just doesn't exist. You're limited to what comes out of the factory.
The sheer number of combinations exposed by the pieces that can number in the thousands for some sets that can be assembled either as the designer intended, completely differently, or even mixed and matched with other pieces from other sets to me feels far more like the parent comment description of exposing the underlying features, shipping a specific vision of them being assembled in a certain way, but allowing alternative visions to be constructed. What you're describing to me is more akin to building blocks; they're larger, more uniform, and only possible to put together in the ways that the designers intended. Stacking them up requires far less effort than putting together a Lego set, and you're not going to have trouble understanding how to stack some pieces so that they're relatively stable, but you're limited by the combinations of ways you can compose uniform cubes. You can't build an arch or bridge because gravity will get in the way, and you don't have the ability to make any other shapes out of the block material.
I'm in this camp as well. The additional machinery required to make library features pluggable often adds a lot of complexity to the user-visible semantics of the language. And it's often not so easy to limit that complexity to only the situations where the user is trying to extend the language.
That's not always true. Sometimes the process of making seemingly core features user-replaceable will reveal simpler, more fundamental abstractions. But this article is a good example of the opposite.
More seriously, I'm not expecting that most people will want to use the underlying language features, nor indeed that those who do should actually do so commonly. They're there to provide for a clean separation between control logic and business logic. And that helps us to create cleaner abstractions so we can test our control logic independently of our business logic.
Most of the time I should be using the control logic we've already written, not writing more of it.
It is not worth it to write better languages if just the "last mile" of final program is more productive --- making new languages is extraordinarily expensive, and this sort of single-shot productivity boost just isn't worth it.
If you can make better libraries --- code reuse and abstractions at hitherto unimaginable ways and then use those libraries to write yet more libraries, you get a tower of abstractions, each yielding every more productivity, like an N stage rocket.
This sort of ever-deepening library ecosystem is scarcely imaginable to most programmers. Or, they think they can imagine it, but everything looks like a leftpad waste of time to them.
I don't think this is possible in a language that needs a runtime, like Go.
I think the reason this is that it can be less work to use compiler magic, and the result is almost (but not quite) as good.
I've been struggling with a related problem over at [1]. Feel free to read this, but it's nowhere near finished. I'm trying to figure out how to do back references cleanly and safely. The basic approach I'm taking is
- We can do just about everything useful with Rc, Weak, RefCell, borrow(), borrow_mut(), upgrade, and downgrade. But it's really wordy and there's a lot of run time overhead. Can we fix the ergonomics, for at least the single-owner case? Probably. The general idea is to be able to write a field access to a weak link as
when what's happening under the hood is - After fixing the ergonomics, can we fix the performance by hoisting some of the checking? Probably. It's possible to check at the drop of sometype whether anybody is using it, strongly or weakly. That allows removing some of the per-reference checking. With compiler support, we can do even more.What I've discovered so far is that the way to write about this is to come up with real-word use cases, then work on the machinery. Otherwise you get lost in type theory. The "Why" has to precede the "How" to get buy-in.
I notice this paper is (2024). Any progress?
[1] https://github.com/John-Nagle/technotes/blob/main/docs/rust/...
[1]: https://plv.mpi-sws.org/rustbelt/ghostcell/
Not clear why it never caught on.
There have been many attempts to solve the Rust back reference problem, but nothing has become popular.
> sometype.upgrade().unwrap().borrow().name
I suspect a hidden `.unwrap()` like that will be highly controversial.
What I'd like to do is move as much of the checking as possible to the drop() of the owning object, and possibly to compile time. If .borrow() calls are very local, it's not too hard to determine that the lifetimes of the borrowed objects don't overlap.
Upgrade is easy to check cheaply at run time for Rc-type cells. Upgrade only fails if the owning object has been dropped. At drop, if weak_count == 0, no dangling weak references outlive the object. If there are more strong references, drop would not be called. With that check, .upgrade() will never fail.
After all, when a programmer codes a potentially fatal .borrow(), they presumably have some reason to be confident the panic won't trigger.
I'm curious: what do the benchmarks say about this?
I wrote https://github.com/Ericson2314/rust-papers a decade ago for a slightly different purpose, but fundamentally we agree.
For those trying to grok their stuff after reading the blog post, consider this.
The borrow checker vs type checker distinction is a hack, a hack that works by relegating a bunch of stuff to be "second class". Second class means that the stuff only occurs within functions, and never across function boundaries.
Proper type theories don't have this "within function, between function" distinction. Just as in the lambda calculus, you can slap a lambda around any term, in "platonic rust" you should be able to get any fragment and make it a reusable abstraction.
The author's here lens is async, which is a good point that since we need to be able to slice apart functions into smaller fragments with the boundaries at await, we need this abstraction ability. With today's Rust in contrast, the only way to do safe manual non-cheating awake would instead to be drasticly limit where one could "await" in practice, to never catch this interesting stuff in action.
In my thing I hadn't considered async at all, but was considering a kind of dual thing. Since these inconsievable types do in fact exist (in a Rust Done Right), and since we can also combine our little functions into a bigger function, then the inescable conclusion is that locations do not have a single fixed type, but have types that vary at different points in the control flow graph. (You can try model the control flow graph as a bunch of small functions and moves, but this runs afowl of non-movable stuff, including borrowed stuff, the ur-non-moveable stuff).
Finally, if we're again trying to make everything first class to have a language without cheating and frustration artificial limits on where abstraction boundaries go, we have to consider not just static locations changing type, but also pointers changing type. (We don't want to liberate some types of locations but not others.) That's where my thing comes in — references that have one type for the pointee at the beginning of the lifetime, and another type at the end.
This stuff might be mind blowing, but if should be seriously pressude. Having second class concepts in the language breeds epiccycles over time. It's how you get C++. Taking the time to make everything first class like this might be scary, but it yields a much more "stable design" that is much more likely to stand the test of time.
All this is about adding things to the language. That's backwards compatible. E.g. Drop doesn't need to be changed, because from every Drop instance a DropVer2 instance can be written instead. async v1 can also continue to exist, just by continuing to generate it's existing shitty unsafe code. And if someone wants something better, they can just use async v2 instead.
People get all freaked out about changing languages, but IMO the FUD is entirely due to sloppy imperative monkey brain. Languages are ideas, and ideas are immutable. The actual question is always, can we do "safe FFI" between two languages. Safe FFI between Rust Edition 20WX and 20YZ is so trivial that people forget to think about it that way. C and C++ is better since C "continues to exist", but of course the bar for "safe FFI" is so low when the language themselves are unsafe within themselves so that safety between them couldn't mean very much.
With harder edition breaks like this, the "safe FFI" mentality actually yields fruit.
But as I investigate Rust, I learn of trivial use cases that cannot be safely represented [0] in Rust's syntax. TFA demonstrates even more provably-safe techniques that are impossible to express safely in Rust. So after all the difficulty of learning Rust, I might still have to choose between performance and safety?
My impression is that Rust 1.91.0 simply isn't a suitable C replacement for many real world use cases. But since it's already being used in production, I worry that backwards compatibility concerns will prevent these issues from being fixed properly, or at all.
Perhaps rust2 will get this right. Until then there's C.
[0]: https://databento.com/blog/why-we-didnt-rewrite-our-feed-han...
Yes it would add like one extra add to every access, but you hardly ever need self-borrows so I think it's probably an acceptable cost in most cases.
So B.method(&self) might do addr(&self.foo) + self.foo, which is stable even if the parent struct A and its raw data field moves?
Then I wonder how to handle the case where the relative &pie reference itself moves. Maybe parsed_data is std::mem::replaced with parsed_data2 (or maybe one of them is an Option<B> and we Option.take() it somewhere else.)
The principles exposed by the language, like lifetimes and mutability, are easy enough to understand. Yet despite understanding them, you get into epic battles with the borrow checker where the new shiny rules you've just learnt turn out to be little help in figuring out why the compiler accepts some code but rejects something similar.
You can't figure it out because these inconceivable types are not written down or explained anywhere. Instead if you are lucky you get examples of what works and what doesn't along with a hand wavy description of why, but usually it's just you versus the compiler error message. Since you don't have a concrete description of what is going to help you reason about what it likely to be accepted, you resort to experimentation to try and find what the compiler does accept. The experimental route takes ages.
If you survive enough bruising rounds of this, you will build up an empirical list of ways of working in Rust. If not, you give up on the language. To be fair, I don't think it takes a huge amount of time in the scheme or things, certainly less time than it takes to know your way around a languages standard library as that can take a month or two. However for those of us with a few languages under our belts, the time it does take comes as a shock. I'm used to it taking a few hours to become acquainted with the syntax and semantics or a new language, not finding myself still having battles with the compiler weeks later.
IMO, if the Rust doco did explain the language things the way this post does, it would make the learning it much easier.