The title is slightly misleading, and so is Linus' response here. This RFC never claimed to be in shape to be immediately mergeable into the mainline kernel as-is. Miguel (the author of the patch) has replied to this mail (and in other places in the thread) that the Rust alloc() is currently called only as a temporary measure to speed up development, and all panic() calls from allocation failures are just as temporary. This is all because the Rust code that hooks into the kernel memory allocation functions is not yet usable.
The main point of this RFC is that "the [in-kernel Rust] support is good enough that prototyping modules can start today." There's no point in making long arguments about alpha-quality design shortcuts on an alpha-quality prototype that are also explicitly mentioned by the patch authors to be of alpha quality.
EDIT: Thanks for the child comments; it seems that Linus is simply not aware of all the specifics and is asking for more information and/or decided to look at the code before reading the full mail thread.
I don't really see anything misleading in the post. Linus says that the comments/responses are from a position of ignorance and it seems like he's just seeking understanding.
If anything is misleading it's linking to random emails in lkml without context. :-P (Maybe that's what you were getting at).
Personally, I'm very excited about Rust in the kernel.
But looks like Linus doesn't know the specifics, he is asking for more info while at the same time making already clear enough that if his concern cannot be addressed than there is no point working on the Rust integration until it is fixed.
Anyone thinking that Rust is ready to be used in the kernel, just read the post first. It is very short. Then read these comments to look for what you need answered.
For me, the main point is Can Rust be written to guarantee that no oom (or other hard fail 128-bit math) panic occurs that is not under control of the written code? I want the answer to be yes and also want to see how (which isn't far off from what Linus is asking).
Yes, easily, OOM is entirely a library created concept in rust, just don't use the standard library (or the `alloc` susbset of the standard library) and you don't have OOMs...
> (or other hard fail 128-bit math)
128 bit math, and floats, need to be avoided by just "not using them", the same as for floats in C code in the kernel...
You got two good answers, but also, there's also now some pressure to make these scenarios better (say, "compile error if you use a i128" rather than "just don't use an i128"), which is nice. I'll be glad on the kernel's needs putting some pressure on Rust to improve.
What I would love to see is Rust in the kernel, with a caveat. That caveat is something akin to Erlang's OTP supervision trees. Erlang has a similar philosophy to Rust in that if something erroneous happens the process fails. The difference is that Erlang OTP is designed to have supervisor processes of the worker processes. The only thing the supervisors do is monitor the processes under them and control when, if, and how those processes are restarted on failure or other conditions. The supervisors themselves have supervisors, right up to the root supervisor for each Erlang application.
Rust got the language and the tooling perfect, but Erlang got the services and service infrastructure perfect. The more I think about it the more I think I should shut up and put up. In other words, apply my knowledge of Erlang and OTP to create a gen_server in Rust as a jumping off point for a OTP like Rust framework, perhaps called OARS (Open Advanced Rust Services). This is definitely bigger than one person. If you'd like to join me on this journey then reply to this comment and I'll send you project details by the end of the weekend.
I don't see how one can run and hide from Linus' point. Exception/panic based work is problematic in libraries, and has no place in kernels. Erlang's OTP supervision trees is as others have pointed out, a runtime issue that app-devs build on. Therefore it's an abstraction that's above the kernel and out of scope w.r.t. to kernel work.
This is a fair and valid point. On reflection, what I am looking for is not something that would be in the kernel. It's more something that is an added layer on Rust, quite possibly with compiler support, to provide for OTP style services and supervisors.
Doing this well requires a heavy enough runtime that it would disqualify it for this kind of work. Not on a technical level, but on a social one. While there have been operating systems created with this sort of runtime, they're not as well known or successful as ones that haven't. I would imagine this email would be a flat "no" if this were a core part of Rust, sadly.
Hey, asking out of pure ignorance - Why do you think this? My naive point of view is that the Linux kernel already has enough of a runtime to support such at hing - a scheduler, kernel threads, and interrupts.
An actor system shouldn't be particularly heavy - you can probably implement an actor in just a couple of bytes.
Agreed that it couldn't be a core part of Rust though.
Maybe it's too off topic or whatever, just curious.
From what I remember the Linux kernel doesn’t even use std.lib. It’s pretty straight up C with no dependencies (the os being low level). Makes kernel module programming difficult.
I don’t understand what you’re getting from this. Crash looping can still happen in OTP, the root supervisor can still die if the crash threshold is met in a small window. This would also be very heavy weight. IIUC the issue is not that errors occur, but that errors occur (and panic) and cannot be handled
Because borrow checking > GC. When you have a GC, you need several times more memory to run the same program with the same performance as without -- and usually, you have to say goodbye to any sort of determinism in execution time, as well.
What Rust brings to the table is guaranteed memory safety without GC, and all memory is released in strictly deterministic time.
So for an OS, it's much better to bring the good bits of Erlang to Rust.
That is a good idea, but one thing I would advise, having both seen several attempts made at this sort of thing and having made one myself [1], try very hard to separate the accidental things Erlang brings to the idea from the fundamental things Erlang brings to the idea. Most attempts I've seen made at this flounder on this pretty hard by trying to port too directly the exact Erlang supervisor tree idea while grinding hard against the rest of the language, rather than porting the core functionality in in a way that integrates natively with the language in question as much as possible.
For instance, one thing I found when I was writing my library that will probably apply to most other languages (probably including Rust) is that Erlang has a somewhat complicated setup step for running a gen_server, with an explicit setup call, a separate execution call, several bits and pieces for 'officially' communicating with a gen_server, etc. But a lot of these things are for dealing with the exact ways that Erlang interacts with processes, and you probably don't need most of them. Simply asking for a process that makes the subprocess "start" from scratch is probably enough, and letting that process use existing communication mechanisms already in the language rather than trying to directly port the Erlang stuff. Similarly, I found no value in trying to provide direct ports of all the different types of gen_server, which aren't so much about the supervision trees (even if that's where they seem to be located) as a set of standard APIs for working with those various things. They're superfluous in a language that already has other solutions for those problems.
In addition to keeping an eye out for features you don't need from Erlang, keep an eye out for features in the host language that may be useful; e.g., the most recent suture integrates with the Go ecosystem's ever-increasing use of context.Contexts as a way to manage termination, which hasn't got a clear Erlang equivalent. (Linking to processes has some overlapping functionality but isn't exactly the same, both offering some additional functionality contexts don't have as well as missing some functionality contexts do have.)
Erlang has a lot of good ideas that I'd love to see ported into more languages. But a lot of attempts to do so flounder on these issues, creating libraries so foreign to the host language that they have zero chance of uptake.
The other thing I'd point out is that even in Go, to say nothing of Rust, crashing is actually fairly uncommon by Erlang standards. Many things that crash in Erlang are statically prevented at compile time in Go, and Rust statically precludes even more of them. However, I have found it OTP-esque supervision trees to be a very nice organizational structure to my code; I use suture in nearly every non-trivial Go program I write because it makes for a really nice modular approach for the question of "how do I start and stop persistent services?". I have seen it hold together runtime services that would otherwise be failing, the way it is supposed to, and that's nice, but the organization structure is still probably the larger benefit.
(There is deep reason for the way Erlang is doing it the way it does, which is that a lot of Erlang's type system, or lack thereof, is for communicating between nodes, so even if you perfectly program Erlang, if two nodes running different versions of code try to communicate with each other and they've changed the protocol you might get a pattern matching fail on the messages flowing between versions. The Erlang way of doing cross-machine communication with this sort of automatic serialization at the language level has not caught on, and all modern languages have a relatively distinct serialization step where this sort of error is better handled, as you try to deserialize the remote message into your internal data structure.)
Anyhow, the upshot is, you want to translate the functionality out of Erlang into other languages, not transliterate it.
I would love to see the same thing, I always told people around me that Erlang/OTP is more akin to an OS than a traditional programming language. That being said, the key feature to enable what Erlang supports is asynchronous termination, which as far as I know is not possible in regular Rust.
I think the current options for killing a generic task in Rust are either 1) make it an async task, which can be cancelled as long as it doesn't accidentally block a thread, or 2) make it a separate process, and have the OS kill it. Do either of those fit this use case?
Aside from Linus' reaction, there are some really interesting pearls in that thread, for example:
Regarding code style[1]:
> The more you make it look like (Kernel) C, the easier it is for us C people to actually read. My eyes have been reading C for almost 30 years by now, they have a lexer built in the optical nerve; reading something that looks vaguely like C but is definitely not C is an utterly painful experience.
> You're asking to join us, not the other way around. I'm fine in a world without Rust.
CoC was already brought into battle[2]:
>> I could be mistaken but you seem angry. Perhaps it wouldn't be a bad idea to read your own code of conduct, I don't think you need a browser for that either.
> Welcome to LKML. CoC does not forbid human emotions just yet. Deal with
it.
These ([3] [4]) messages have an interesting perspective about maintenance.
> I'm sure about one thing, the C bugs we have today will be
fixable in 20 years. I'm not even sure the Rust code we'll merge today
will still be compilable in 10 years nor will support the relevant
architectures available by then, and probably this code will have to
be rewritten in C to become maintained again.
Linus wants to see a real, working kernel driver instead of Android Binder[5]:
>Would there be some kind of real driver or something that people could
use as a example of a real piece of code that actually does something
meaningful?
> I don't see how the two languages might coexist peacefully without rust toolchain being necessary for building any kernel useful in practice and anyone seriously involved in kernel development having to be proficient in both languages.
I can empathise with that. Just last week I butted head with an issue in a python package that requires Rust internally. The lib compiles fine on its own, but something gets screwed when running in a virtualenv. Opened a bug in github, and nobody has any idea about how to get even a detailed log out of the rust toolchain.
I'm sympathetic about Rust, I really am. But sprinkling it mindlessly everywhere is a big risk.
> The more you make it look like (Kernel) C, the easier it is for us C people to actually read. My eyes have been reading C for almost 30 years by now, they have a lexer built in the optical nerve; reading something that looks vaguely like C but is definitely not C is an utterly painful experience.
I think he makes a good point about the fact that it's certainly possible the Rust code written today won't still compile in 10 years, but writing Rust in C-style seems like a terrible approach. Write using the idioms of the language used.
I agree, but if it helps creating more and better drivers, it might be a good thing considering the lifetime of the target hardware matches Rust lifetime.
>>> I could be mistaken but you seem angry. Perhaps it wouldn't be a bad idea to read your own code of conduct
A passive-aggressive "you mad bro?" followed by namechecking the coc, all in service of doubling down on antagonizing someone over their choice in workflow? Good grief.
“CoC” seems to suffer from the same problem that “bad cops” and “zero tolerance” politicians seem to suffer from, so that's why it's often brought up.
It seems to be a common thing that those that demand the strictest morality rules also seem to have the most aggression problems and often overstep their own rules, but typically have an excuse ready why in their case it's different.
Though, perhaps it simply stands out more if it's one to chant “code of conduct” or “zero tolerance”, but as far as statistics in the Dutch parliament goes, it's often pointed out that all the parties that are in favor of lighter punishments and rehabilitation tend to be spot free, whereas politicians of parties that advocate harsh punishments and zero tolerance tend to very often have past criminal records themselves.
Eh I'm normally against bludgeoning people with a CoC, but the guy he responded to was basically telling him to "go fuck yourself" masked by euphemisms. So I think this was just a passive-aggressive response in kind to a rude email.
I've yet to see a program that renders HTML (including all the cruft often used in docs, which might include SVG graphics and whatnot) sanely in ASCII. Lynx does not qualify, it's output is atrocious crap.
Yes, lynx lets you read HTML in ASCII, but at the cost of bleeding eyeballs and missing content.
Nothing beats a sane ASCII document with possibly, where really needed some ASCII art.
Sadly the whole kernel documentation project is moving away from that as well, which just means I'm back to working on an undocumented codebase. This rst crap they adopted is unreadable garbage.
Rust's strengths lie where Rust is written like Rust. If Linus wants C, he knows where to find it. Rust isn't for "C people" -- it's for their replacements.
- most such interfaces do have a constant overhead that all programmers using the language have to pay.
- some major operating systems (like Linux, ehem) make these interfaces useless for all their user space apps by enabling overcommit by default, which makes it hard/impossible to write portable code that can handle OOM
Explicit OOM handling would mean that most users end up paying a relatively high upfront cost for something that in practice for them (e.g. if they are Linux programmers) delivers no value.
For example, on Linux with overcommit (the default), even if the system is out of memory, malloc won't return null. It returns a pointer that's not null, such that your if(ptr == nullptr) will act as if everything is "ok", but then, when you try to read/write that memory, that will trigger a hardware exception, that the kernel will catch, and then the kernel will tell the OOM-killer to "make space" by killing "some process", and maybe your app is just killed, or some other app that your app is working with is killed leading to a race condition, or... or....
When your app is killed by the OOM killer, the signal that your app gets is "irrecoverable", which means that your app will die, you can at best try to do some cleanup before it does, but it will die nevertheless.
So I find it extremely ironic for Linus to argue that practical programming languages are hard to use for environments that must handle OOM errors, when they are championing one of the major platforms that makes handling OOM useless by default.
I keep saying this, but the obvious fix is for the Linux kernel to use overcommit internally just like they expect user space to do. When the kernel then runs out of memory, it should then start killing drivers at random to make for some space. If they think that's such a great default behavior, they should commit to it. \s
The "but the OOM killer!" argument has been a disaster for Rust, and keeps derailing all design discussions.
• Linux is not the only OS in the world.
• Even on Linux you have containers/cgroups that can impose hard limits.
• Platforms without virtual memory are also an important target for Rust.
• On 32-bit platforms you can run out of address space before you run out of RAM.
• Regardless of what the OS does, the application may still want to impose its own internal limit (e.g. https://lib.rs/cap) to avoid being OOM-killed or swap death.
People keep telling me how Linux never runs out of memory, while I'm currently firefighting a torrent of coredumps caused by Rust's self-own on OOM that actually happens.
I generally love Rust, but its OOM handling is awful, and the "but the OOM killer" nonsense is to blame for stalling the absolutely critical fixes it urgently needs.
> For example, on Linux with overcommit (the default), even if the system is out of memory, malloc won't return null. It returns a pointer that's not null, such that your if(ptr == nullptr) will act as if everything is "ok",
Not always. The default overcommit policy will reject certain allocations that are too big. Also, you can change the overcommit policy if you need it.
Also, overcommit policy not rejecting allocations by default tends to be more useful than harmful. One example is forking processes: the memory space gets duplicated with copy-on-write, so if you didn't allow overcommit you could end up with processes that couldn't be forked due to that behavior.
Not to mention that most programs don't use all the memory they have assigned, so without overcommit you'd have a lot of problems with RAM underusage.
> So I find it extremely ironic for Linus to argue that practical programming languages are hard to use for environments that must handle OOM errors, when they are championing one of the major platforms that makes handling OOM useless by default.
Kernel and user space programming are very, very different. A lot of developers use languages with garbage collection without issues, but that's not acceptable in kernel programming. The fact that the platform provides X feature does not mean it has to be developed including X feature. In this case it's very clear. Even if they used overcommit in kernel code, Linux does not always overcommit memory, so you'd still need to manage the OOM case in kernel properly.
So, basically you are saying "Rust is mostly used as an application programming language anyways, so let's not pester those users with the overhead of OOM handling"?
Fair enough. But this would mean that Rust is (factually) a systems language in the same sense that Go was initially declared to be a systems language.
If I look at the Rust community (I give it a try from time to time), I would totally agree with your point, that most users would be pestered by this overhead.
I see mostly CLI tools, "web apps" and the kind.
Furthermore, I just can't shake off the feeling that the embedded (bare metal, 16 or 32 bit architectures) or (in-house) kernel crowd will always be 2nd class citizens.
I keep saying this, but the obvious fix is for the Linux kernel to use overcommit internally
just like they expect user space to do.
I'm not a kernel developer, but aren't you (maybe) asking for a bit too much? You ask the Linux kernel devs to change the kernel in a significant way, just so you can use Rust for writing drivers in a way you write user space applications.
You can't solve memory allocation problems by turning off overcommit and handling alloc errors in rust. You don't get Null, you get panic.
That's a problem for the kernel, even if you argue it's not a problem elsewhere.
Start killing drivers is nuts, oh look my screen went blank, and three HDs stopped working. Damn where is my swap.
It's not lacking, the standard library just defaults to not use it by default because handling OOM is a conceptual mess (yes, the OOM handling features C/C++ has are included in this).
But even in the standard library you have methods (some not stable) like `try_reserve` which returns an error is allocation fails.
Anyway all allocation parts are part of the standard library (or the alloc library), i.e. they are not a core part of the language itself.
I don't think anyone ever planed to use rust's standard library as-it-is in the kernel (it's just not designed for this, e.g. see panics) or pull in external dependencies from cargo/crates.io without vendoring them.
So it's totally possible to run rust without alloc caused panics.
Now besides that there is the question about panics outside of allocations. Like e.g. if code realizes it ran into a violated invariant, e.g. it knows we have a bug which isn't explicitly handled.
And guess what the kernel already has a handler for it: `BUG()` so basically any panic will call `BUG()` (and we don't panic on memory allocations).
Now maybe some tweaks to the panic system are necessary to make sure this all works well (panic=BUG(), is like panic=abort and on abort call BUG()!).
Now wrt. the integer and float parts, there are two thinks first the panic will likely call without debug assertions like overflow checks (which would call BUG()!! So many are gone, but here are some special cases around floats and 128bit integers on platforms which don't support it where calling BUG() is inappropriate but I have to look into this, tbh.
So short:
- panic on mem-alloc failure is a (lib)std/alloc thing, kernel code would anyway have used something easel
- panic in the kernel can be made into being basically BUG()
- issues around float and 128-bit integers should be fixable, at worst with a compiler flag. But embedded code is by it affected, too. So there is a good chance there is already a fix.
I agree Rust should have it, but it is VERY hard to do correctly.
I've worked on some systems which claimed they were dealing with it, but when I purposefully pushed it (by making a malloc which would occasionally fail), I quickly uncovered dozens of bugs, and the solution was to stop pretending would could sensibly handle low-memory situations. SQLite famously does handle this situation correctly, but it is a huge amount of work.
There are counter examples. For example, data processing as part of a measurement application: If allocating the data fails, the application should abort that, but keep running to allow further control and let the user reduce e.g. the sampling size.
I know of one application with a worldwide customer base that supports this.
Rust std::alloc has the same standard interface as everything else- return null on failure to allocate. It's just that the development in std collections to use this is in progress. And it's being retrofitted to collections that assume they can just abort or panic on alloc failure.
Out of memory means, your system is simply not designed for the task at hand.
The kernel returning -ENOMEM only masks the fact that eventually Linux will have to OOM Panic if it can't OOM Kill. Hell imagine the swapping and the I/O spike because your VFS cache has been or is currently being purged.
I honestly think the best case is to just fail when a fundamental resource is simply not there.
> I honestly think the best case is to just fail when a fundamental resource is simply not there.
Indeed. I'm using Firefox in a VM. It's a pain, because invariably at some point Firefox uses enough memory that the VM starts to swap. And then the whole thing grinds to a halt, and I usually just end up with the VM equivalent of power-off.
Instead I'd be perfectly fine with the kernel telling Firefox "computer says no" when it tries to malloc, before the system runs out of memory, and then Firefox can do whatever it wants with that.
Yes I know there's some cgroups magic or whatever I can do, but man, why does it have to be so painful?
> Out of memory means, your system is simply not designed for the task at hand.
Speaking as somebody who's never built such a system, wouldn't maintaining some opportunistic cache (e.g. for some space-time-tradeoff) be a valid use case of asking for more memory and gracefully degrading in case it's not available?
Error handling would simply consist of not expanding the cache size (if it happens during an allocation related to the cache) or freeing up some cache memory (if it happens for an essential allocation).
> Out of memory means, your system is simply not designed for the task at hand.
You seem not very familiar with the memory "management" behavior of Linux and the wonderful ecosystem it has engendered. For example no matter how much physical memory you have, Chrome for example will just crash all the time if you turn off Linux's insane "lie about memory allocation succeeding" default.
This seems to be a non-issue? Torvalds has valid concerns, the patch submitter acknowledges those concerns and describes how they can and will be fixed before anything is merged.
We are lucky that this is a link to the mailing list, and you can read the answers. Tech "journalists" read all mails from Linus and try to create click-baity articles from them, completely ignoring the context.
It's interesting tech stuff that's relevant to HN, it certainly caught my interest. There doesn't have to be a conflict for something to be a good HN submission.
In the embedded Rust world it is common to explicitly define a custom global allocator and panic handler. The global allocator is even optional although you then won't be able to use heap allocated structs like String and Vec. Therefore the Rust compiler does not force you to use the built in allocator or panic handler and you are free to implement them how you choose.
As a C developer for embedded, considering Rust for a long time now, the panic thing is something that bothers me. I don't want/I can't panic. I want to be returned false, null or whatever.
Do I have to check if an external crate that I am using would panic? If so, how do I prevent the crate from panicking?
From my perspective, Rust is kind-of designed to support code like this (non real Rust code follows):
What if some of those calls fail? How do I detect an error? Alright, it might panic, but what if I have to keep going forward, even in case of an error, in my embedded application? Should I split the different function calls and check for errors?
Kernel development intersects with embedded development in many points. I'm sure I am not the only one with these doubts.
Calls that fail return `Result<OkType, ErrorType>`. Here's something like what `unwrap()` does (actual implementation may have some nuances I didn't capture here):
So, if you don't accept panic, don't call `unwrap()` unless you're very sure that the returned value couldn't fail in the way you're using the API. You could think of `unwrap()` in Rust as semantically similar to `assert(error == 0)` in C. Avoid or use in the same places you would avoid or use that assertion in C.
That means you must handle the Err case, often by just returning it to the caller (like C). The ? operator is syntax sugar for this: return if err, or give me the ok value if non-err.
fn my_fn() -> Result<(), Error> {
// if this failed, returns some Err()
let my_x = my_struct.do_something_that_can_fail()?;
// ditto
let my_y = my_x.something_else_that_can_fail()?;
// no failures? Ok
Ok(my_y)
}
Of course, these can be chained with the exact same characteristics:
If the error type of the result differs you can use map_err or write a converter by implementing the From trait for your the custom error struct you want to map to. The one thing you can't find out is if the calling function will panic in its function or a function that it calls. When you write embedded libraries you are never supposed to panic but there is nothing to enforce this which is, admittedly, not ideal.
It would really panic if it ran into an unhandled condition that you explicitly sent it to or left as is. What I mean is pretend you have some external reading/data coming in and you parse it to have the field "name" to always have a value of "scoutt".
match data.name {
"scoutt" => return True,
_ => panic!(),
}
In that case, because you "know" that the name field cannot be anything but your specific value ,you don't really mind the matching pattern and Rust is satisfied. But then you get a runtime panic! because all of the sudden the value changed to "joe".
Long story short is as long as you handle and declare ahead of time the proper fail conditions and paths to take at compile, you really shouldn't come across a panic.
It seems a serious shortcoming if you can't allocate String or Vec using a custom allocator? Presumably you can pass an arena parameter to create the custom allocator?
There's plenty of room between <unlimited heap> and <preallocated>, I don't really understand why, e.g., you couldn't have a pool for a particular usage of String, or fixed size objects?
Also, introspection to ask which pool an object is allocated on, how much space is available (units for fixed size, largest hole for variable size)?
Is it just a syntax choice or is there a rusty reason not to have a thread-local list of allocators?
It's just time; we have stabilized the interface for the single global allocator, but the more general allocation API is still being worked on.
You can do all of these things for your own data structures, but the ones in liballoc shipped without that support, because we had to get Rust 1.0 out the door. Now support is being retrofitted, and that's part of why doing so was okay before; the plan to do so in a reasonable way existed at stabilization time.
There's nothing special about String other than it's in the standard library. It doesn't even have special cased syntax (&str does have special syntax, it doesn't allocate).
String does need to know how to deallocate itself, so the compiler has to know what allocated it. It can be moved between threads, so a thread local allocator doesn't work.
If you want a String type which is allocated differently, or which stores a pointer to its deallocate, you can easily implement that as a new type by hand.
(The standard library types might some day become generic over allocator, which would allow statically using a different allocator with the same api. The compiler would then force you to keep track of which allocator your string was allocated with).
I see, so anything that can possibly fail should return a result instead of "maybe" panicking as a side effect? So no panics at all in the language? That may adversely affect the ergonomics of the language for user space applications. Perhaps it would be better to build a completely fallible version of the standard library and use linting to enforce no panics.
I don't have a problem with the idea of panic, but the way it's used seems like a wart. In a language with a result type why there are there any cases where the system isn't too broken to return a result, but default to panicking anyway?
Thinking about it from the embedded side, my panic is going to tell the failsafe circuitry that the processor can't be trusted. That's too aggressive of an action for any case where we still believe that the processor can return control to a caller that could have a more-graceful error path.
I see in that document the mention of a libredox as a libstd replacement, I'm guessing that means they have their own primitives with constructors that handle allocation failures?
Readit News
The main point of this RFC is that "the [in-kernel Rust] support is good enough that prototyping modules can start today." There's no point in making long arguments about alpha-quality design shortcuts on an alpha-quality prototype that are also explicitly mentioned by the patch authors to be of alpha quality.
See e.g. https://lkml.org/lkml/2021/4/14/1130 and https://lkml.org/lkml/2021/4/14/1023
EDIT: Thanks for the child comments; it seems that Linus is simply not aware of all the specifics and is asking for more information and/or decided to look at the code before reading the full mail thread.
If anything is misleading it's linking to random emails in lkml without context. :-P (Maybe that's what you were getting at).
Personally, I'm very excited about Rust in the kernel.
Why?
For me, the main point is Can Rust be written to guarantee that no oom (or other hard fail 128-bit math) panic occurs that is not under control of the written code? I want the answer to be yes and also want to see how (which isn't far off from what Linus is asking).
Yes, easily, OOM is entirely a library created concept in rust, just don't use the standard library (or the `alloc` susbset of the standard library) and you don't have OOMs...
> (or other hard fail 128-bit math)
128 bit math, and floats, need to be avoided by just "not using them", the same as for floats in C code in the kernel...
Rust got the language and the tooling perfect, but Erlang got the services and service infrastructure perfect. The more I think about it the more I think I should shut up and put up. In other words, apply my knowledge of Erlang and OTP to create a gen_server in Rust as a jumping off point for a OTP like Rust framework, perhaps called OARS (Open Advanced Rust Services). This is definitely bigger than one person. If you'd like to join me on this journey then reply to this comment and I'll send you project details by the end of the weekend.
Hey, asking out of pure ignorance - Why do you think this? My naive point of view is that the Linux kernel already has enough of a runtime to support such at hing - a scheduler, kernel threads, and interrupts.
An actor system shouldn't be particularly heavy - you can probably implement an actor in just a couple of bytes.
Agreed that it couldn't be a core part of Rust though.
Maybe it's too off topic or whatever, just curious.
(I also don't really think the linux kernel people would be interested...)
In fact, you can run Erlang directly on raw metal, or write an OS in Erlang: http://www.erlang-factory.com/static/upload/media/1498583896...
What Rust brings to the table is guaranteed memory safety without GC, and all memory is released in strictly deterministic time.
So for an OS, it's much better to bring the good bits of Erlang to Rust.
For instance, one thing I found when I was writing my library that will probably apply to most other languages (probably including Rust) is that Erlang has a somewhat complicated setup step for running a gen_server, with an explicit setup call, a separate execution call, several bits and pieces for 'officially' communicating with a gen_server, etc. But a lot of these things are for dealing with the exact ways that Erlang interacts with processes, and you probably don't need most of them. Simply asking for a process that makes the subprocess "start" from scratch is probably enough, and letting that process use existing communication mechanisms already in the language rather than trying to directly port the Erlang stuff. Similarly, I found no value in trying to provide direct ports of all the different types of gen_server, which aren't so much about the supervision trees (even if that's where they seem to be located) as a set of standard APIs for working with those various things. They're superfluous in a language that already has other solutions for those problems.
In addition to keeping an eye out for features you don't need from Erlang, keep an eye out for features in the host language that may be useful; e.g., the most recent suture integrates with the Go ecosystem's ever-increasing use of context.Contexts as a way to manage termination, which hasn't got a clear Erlang equivalent. (Linking to processes has some overlapping functionality but isn't exactly the same, both offering some additional functionality contexts don't have as well as missing some functionality contexts do have.)
Erlang has a lot of good ideas that I'd love to see ported into more languages. But a lot of attempts to do so flounder on these issues, creating libraries so foreign to the host language that they have zero chance of uptake.
The other thing I'd point out is that even in Go, to say nothing of Rust, crashing is actually fairly uncommon by Erlang standards. Many things that crash in Erlang are statically prevented at compile time in Go, and Rust statically precludes even more of them. However, I have found it OTP-esque supervision trees to be a very nice organizational structure to my code; I use suture in nearly every non-trivial Go program I write because it makes for a really nice modular approach for the question of "how do I start and stop persistent services?". I have seen it hold together runtime services that would otherwise be failing, the way it is supposed to, and that's nice, but the organization structure is still probably the larger benefit.
(There is deep reason for the way Erlang is doing it the way it does, which is that a lot of Erlang's type system, or lack thereof, is for communicating between nodes, so even if you perfectly program Erlang, if two nodes running different versions of code try to communicate with each other and they've changed the protocol you might get a pattern matching fail on the messages flowing between versions. The Erlang way of doing cross-machine communication with this sort of automatic serialization at the language level has not caught on, and all modern languages have a relatively distinct serialization step where this sort of error is better handled, as you try to deserialize the remote message into your internal data structure.)
Anyhow, the upshot is, you want to translate the functionality out of Erlang into other languages, not transliterate it.
[1]: https://github.com/thejerf/suture
Deleted Comment
Dead Comment
Regarding code style[1]:
> The more you make it look like (Kernel) C, the easier it is for us C people to actually read. My eyes have been reading C for almost 30 years by now, they have a lexer built in the optical nerve; reading something that looks vaguely like C but is definitely not C is an utterly painful experience.
> You're asking to join us, not the other way around. I'm fine in a world without Rust.
CoC was already brought into battle[2]:
>> I could be mistaken but you seem angry. Perhaps it wouldn't be a bad idea to read your own code of conduct, I don't think you need a browser for that either.
> Welcome to LKML. CoC does not forbid human emotions just yet. Deal with it.
These ([3] [4]) messages have an interesting perspective about maintenance.
> I'm sure about one thing, the C bugs we have today will be fixable in 20 years. I'm not even sure the Rust code we'll merge today will still be compilable in 10 years nor will support the relevant architectures available by then, and probably this code will have to be rewritten in C to become maintained again.
Linus wants to see a real, working kernel driver instead of Android Binder[5]:
>Would there be some kind of real driver or something that people could use as a example of a real piece of code that actually does something meaningful?
[1] https://lkml.org/lkml/2021/4/16/118
[2] https://lkml.org/lkml/2021/4/16/143
[3] https://lkml.org/lkml/2021/4/16/181
[4] https://lkml.org/lkml/2021/4/16/283
[5] https://lkml.org/lkml/2021/4/14/1091
> I don't see how the two languages might coexist peacefully without rust toolchain being necessary for building any kernel useful in practice and anyone seriously involved in kernel development having to be proficient in both languages.
I can empathise with that. Just last week I butted head with an issue in a python package that requires Rust internally. The lib compiles fine on its own, but something gets screwed when running in a virtualenv. Opened a bug in github, and nobody has any idea about how to get even a detailed log out of the rust toolchain.
I'm sympathetic about Rust, I really am. But sprinkling it mindlessly everywhere is a big risk.
I think he makes a good point about the fact that it's certainly possible the Rust code written today won't still compile in 10 years, but writing Rust in C-style seems like a terrible approach. Write using the idioms of the language used.
It’s possible but unlikely. The Editions feature [1] has been specifically designed to provide longevity.
[1]: https://doc.rust-lang.org/edition-guide/editions/index.html
A passive-aggressive "you mad bro?" followed by namechecking the coc, all in service of doubling down on antagonizing someone over their choice in workflow? Good grief.
It seems to be a common thing that those that demand the strictest morality rules also seem to have the most aggression problems and often overstep their own rules, but typically have an excuse ready why in their case it's different.
Though, perhaps it simply stands out more if it's one to chant “code of conduct” or “zero tolerance”, but as far as statistics in the Dutch parliament goes, it's often pointed out that all the parties that are in favor of lighter punishments and rehabilitation tend to be spot free, whereas politicians of parties that advocate harsh punishments and zero tolerance tend to very often have past criminal records themselves.
I've yet to see a program that renders HTML (including all the cruft often used in docs, which might include SVG graphics and whatnot) sanely in ASCII. Lynx does not qualify, it's output is atrocious crap.
Yes, lynx lets you read HTML in ASCII, but at the cost of bleeding eyeballs and missing content.
Nothing beats a sane ASCII document with possibly, where really needed some ASCII art.
Sadly the whole kernel documentation project is moving away from that as well, which just means I'm back to working on an undocumented codebase. This rst crap they adopted is unreadable garbage.
It has never been an issue in my use case of Rust, but the lack of an interface for when the system is out of memory is problematic.
Not only in kernel, but in all system-level software.
Then, it happens, less and less frequently, but still, I want to know and I want to be able to handle it.
- most such interfaces do have a constant overhead that all programmers using the language have to pay.
- some major operating systems (like Linux, ehem) make these interfaces useless for all their user space apps by enabling overcommit by default, which makes it hard/impossible to write portable code that can handle OOM
Explicit OOM handling would mean that most users end up paying a relatively high upfront cost for something that in practice for them (e.g. if they are Linux programmers) delivers no value.
For example, on Linux with overcommit (the default), even if the system is out of memory, malloc won't return null. It returns a pointer that's not null, such that your if(ptr == nullptr) will act as if everything is "ok", but then, when you try to read/write that memory, that will trigger a hardware exception, that the kernel will catch, and then the kernel will tell the OOM-killer to "make space" by killing "some process", and maybe your app is just killed, or some other app that your app is working with is killed leading to a race condition, or... or....
When your app is killed by the OOM killer, the signal that your app gets is "irrecoverable", which means that your app will die, you can at best try to do some cleanup before it does, but it will die nevertheless.
So I find it extremely ironic for Linus to argue that practical programming languages are hard to use for environments that must handle OOM errors, when they are championing one of the major platforms that makes handling OOM useless by default.
I keep saying this, but the obvious fix is for the Linux kernel to use overcommit internally just like they expect user space to do. When the kernel then runs out of memory, it should then start killing drivers at random to make for some space. If they think that's such a great default behavior, they should commit to it. \s
• Linux is not the only OS in the world.
• Even on Linux you have containers/cgroups that can impose hard limits.
• Platforms without virtual memory are also an important target for Rust.
• On 32-bit platforms you can run out of address space before you run out of RAM.
• Regardless of what the OS does, the application may still want to impose its own internal limit (e.g. https://lib.rs/cap) to avoid being OOM-killed or swap death.
People keep telling me how Linux never runs out of memory, while I'm currently firefighting a torrent of coredumps caused by Rust's self-own on OOM that actually happens.
I generally love Rust, but its OOM handling is awful, and the "but the OOM killer" nonsense is to blame for stalling the absolutely critical fixes it urgently needs.
Not always. The default overcommit policy will reject certain allocations that are too big. Also, you can change the overcommit policy if you need it.
Also, overcommit policy not rejecting allocations by default tends to be more useful than harmful. One example is forking processes: the memory space gets duplicated with copy-on-write, so if you didn't allow overcommit you could end up with processes that couldn't be forked due to that behavior. Not to mention that most programs don't use all the memory they have assigned, so without overcommit you'd have a lot of problems with RAM underusage.
> So I find it extremely ironic for Linus to argue that practical programming languages are hard to use for environments that must handle OOM errors, when they are championing one of the major platforms that makes handling OOM useless by default.
Kernel and user space programming are very, very different. A lot of developers use languages with garbage collection without issues, but that's not acceptable in kernel programming. The fact that the platform provides X feature does not mean it has to be developed including X feature. In this case it's very clear. Even if they used overcommit in kernel code, Linux does not always overcommit memory, so you'd still need to manage the OOM case in kernel properly.
Fair enough. But this would mean that Rust is (factually) a systems language in the same sense that Go was initially declared to be a systems language.
If I look at the Rust community (I give it a try from time to time), I would totally agree with your point, that most users would be pestered by this overhead. I see mostly CLI tools, "web apps" and the kind.
Furthermore, I just can't shake off the feeling that the embedded (bare metal, 16 or 32 bit architectures) or (in-house) kernel crowd will always be 2nd class citizens.
I'm not a kernel developer, but aren't you (maybe) asking for a bit too much? You ask the Linux kernel devs to change the kernel in a significant way, just so you can use Rust for writing drivers in a way you write user space applications.You can change the Linux memory overcommit policy using sysctls if you need it.
Deleted Comment
It's not lacking, the standard library just defaults to not use it by default because handling OOM is a conceptual mess (yes, the OOM handling features C/C++ has are included in this).
But even in the standard library you have methods (some not stable) like `try_reserve` which returns an error is allocation fails.
Anyway all allocation parts are part of the standard library (or the alloc library), i.e. they are not a core part of the language itself.
I don't think anyone ever planed to use rust's standard library as-it-is in the kernel (it's just not designed for this, e.g. see panics) or pull in external dependencies from cargo/crates.io without vendoring them.
So it's totally possible to run rust without alloc caused panics.
Now besides that there is the question about panics outside of allocations. Like e.g. if code realizes it ran into a violated invariant, e.g. it knows we have a bug which isn't explicitly handled.
And guess what the kernel already has a handler for it: `BUG()` so basically any panic will call `BUG()` (and we don't panic on memory allocations).
Now maybe some tweaks to the panic system are necessary to make sure this all works well (panic=BUG(), is like panic=abort and on abort call BUG()!).
Now wrt. the integer and float parts, there are two thinks first the panic will likely call without debug assertions like overflow checks (which would call BUG()!! So many are gone, but here are some special cases around floats and 128bit integers on platforms which don't support it where calling BUG() is inappropriate but I have to look into this, tbh.
So short:
- panic on mem-alloc failure is a (lib)std/alloc thing, kernel code would anyway have used something easel
- panic in the kernel can be made into being basically BUG()
- issues around float and 128-bit integers should be fixable, at worst with a compiler flag. But embedded code is by it affected, too. So there is a good chance there is already a fix.
I've worked on some systems which claimed they were dealing with it, but when I purposefully pushed it (by making a malloc which would occasionally fail), I quickly uncovered dozens of bugs, and the solution was to stop pretending would could sensibly handle low-memory situations. SQLite famously does handle this situation correctly, but it is a huge amount of work.
I know of one application with a worldwide customer base that supports this.
Deleted Comment
Out of memory means, your system is simply not designed for the task at hand. The kernel returning -ENOMEM only masks the fact that eventually Linux will have to OOM Panic if it can't OOM Kill. Hell imagine the swapping and the I/O spike because your VFS cache has been or is currently being purged. I honestly think the best case is to just fail when a fundamental resource is simply not there.
Indeed. I'm using Firefox in a VM. It's a pain, because invariably at some point Firefox uses enough memory that the VM starts to swap. And then the whole thing grinds to a halt, and I usually just end up with the VM equivalent of power-off.
Instead I'd be perfectly fine with the kernel telling Firefox "computer says no" when it tries to malloc, before the system runs out of memory, and then Firefox can do whatever it wants with that.
Yes I know there's some cgroups magic or whatever I can do, but man, why does it have to be so painful?
[1]: https://ziglang.org/learn/why_zig_rust_d_cpp/#no-hidden-allo...
Speaking as somebody who's never built such a system, wouldn't maintaining some opportunistic cache (e.g. for some space-time-tradeoff) be a valid use case of asking for more memory and gracefully degrading in case it's not available?
Error handling would simply consist of not expanding the cache size (if it happens during an allocation related to the cache) or freeing up some cache memory (if it happens for an essential allocation).
You seem not very familiar with the memory "management" behavior of Linux and the wonderful ecosystem it has engendered. For example no matter how much physical memory you have, Chrome for example will just crash all the time if you turn off Linux's insane "lie about memory allocation succeeding" default.
Deleted Comment
See for alloc: https://docs.rust-embedded.org/book/collections/index.html See for panicking: https://docs.rust-embedded.org/book/start/panicking.html
Do I have to check if an external crate that I am using would panic? If so, how do I prevent the crate from panicking?
From my perspective, Rust is kind-of designed to support code like this (non real Rust code follows):
What if some of those calls fail? How do I detect an error? Alright, it might panic, but what if I have to keep going forward, even in case of an error, in my embedded application? Should I split the different function calls and check for errors?Kernel development intersects with embedded development in many points. I'm sure I am not the only one with these doubts.
That means you must handle the Err case, often by just returning it to the caller (like C). The ? operator is syntax sugar for this: return if err, or give me the ok value if non-err.
Of course, these can be chained with the exact same characteristics:match data.name { "scoutt" => return True, _ => panic!(),
}
In that case, because you "know" that the name field cannot be anything but your specific value ,you don't really mind the matching pattern and Rust is satisfied. But then you get a runtime panic! because all of the sudden the value changed to "joe".
Long story short is as long as you handle and declare ahead of time the proper fail conditions and paths to take at compile, you really shouldn't come across a panic.
There's plenty of room between <unlimited heap> and <preallocated>, I don't really understand why, e.g., you couldn't have a pool for a particular usage of String, or fixed size objects?
Also, introspection to ask which pool an object is allocated on, how much space is available (units for fixed size, largest hole for variable size)?
Is it just a syntax choice or is there a rusty reason not to have a thread-local list of allocators?
You can do all of these things for your own data structures, but the ones in liballoc shipped without that support, because we had to get Rust 1.0 out the door. Now support is being retrofitted, and that's part of why doing so was okay before; the plan to do so in a reasonable way existed at stabilization time.
String does need to know how to deallocate itself, so the compiler has to know what allocated it. It can be moved between threads, so a thread local allocator doesn't work.
If you want a String type which is allocated differently, or which stores a pointer to its deallocate, you can easily implement that as a new type by hand.
(The standard library types might some day become generic over allocator, which would allow statically using a different allocator with the same api. The compiler would then force you to keep track of which allocator your string was allocated with).
Thinking about it from the embedded side, my panic is going to tell the failsafe circuitry that the processor can't be trusted. That's too aggressive of an action for any case where we still believe that the processor can return control to a caller that could have a more-graceful error path.