Readit News logoReadit News
Animats · 9 years ago
Very nice. It was time to do this.

There are three near-term products to develop on this:

- A small router (home/small office, not data center)

- A DNS server

- A BGP server

Those are standalone boxes which do specific jobs, they're security-critical, and the existing implementations have all had security problems. We need tougher systems in those areas.

nickpsecurity · 9 years ago
I was worried a whole UNIX might be too big a project so I gave him old school recommendations before. Essentially told him to make a split system that lets you run on tiny kernel bare metal, run UNIXy apps with different API, and let them communicate through some IPC mechanism. That way, like security & MILS kernels, people can write security critical components in isolated partitions with checks at interfaces. Already done with Ada and embedded Java. Rust is best one to try it with now.

Far as your recommendations, I like those. I'll add a web server, Redox or UNIX compatible, that's efficient enough to be deployed in all these web-enabled embedded devices. The dynamic part can just be Rust plugins or something. However, just a robust Ethernet stack, networking stack, time API, and simple filesystem could be used to implement all of yours, the web server, and more. So, I encourage people building these OS projects to stay on 80/20 rule hitting features almost all critical things use. Others can jump in at application layer from there.

Animats · 9 years ago
A web server gets big, complicated, has lots of add-on parts, and has performance constraints. Small routers, DNS servers, and BGP servers are small, closed systems that should Just Work. You want to get them working, lock the code into read-only memory, and forget them.
burfog · 9 years ago
In other words, like GNU HURD. That design is very very hard to make correct and fast.

Correctness suffers because the UNIX API has all sorts of interactions between different parts. This includes atomicity. It's a bear to get this right with IPC.

Performance suffers because you are unable to effectively share data structures. This too relates to the interactions between different parts of the UNIX API.

Look, there is a reason GNU HURD is slow and suffers from incompatibility. It's a cute thought experiment, dominating academia around 1990, but it's not actually fast or maintainable. Experience has proven this.

polyfill · 9 years ago
This may be helpful for the DNS server https://github.com/gtadam/rust-dns
bluejekyll · 9 years ago
I'm working on something: https://github.com/bluejekyll/trust-dns

I'm in the midst of implementing DNSSec right now, I've got RRSIG and DNSKEY validation back to the root now. I'm working on negative query validation right now, NSEC and NSEC3. Maybe a new release in a few weeks after that is supported.

nickpsecurity · 9 years ago
Interesting. So will this for anyone brave enough to use SPARK and Rust:

http://ironsides.martincarlisle.com/

IRONSIDES uses SPARK to show freedom of exceptions or single-packet DOS. A translation to Rust would probably preserve most of those properties. I also thought a parallel development between Ada2012/SPARK and Rust would be interesting to see if one set of checkers catch something another misses in same piece of software.

qz_ · 9 years ago
I'm pretty impressed that 37 people are capable of writing an entire OS with a gui in under a year. Looks really cool.
teamhappy · 9 years ago
Operating systems aren't that big, and if you know your stuff, I'm sure it's not too hard to pull off. Here's an example I borrowed from StackExchange:

According to cloc run against 3.13, Linux is about 12 million lines of code. 7 million LOC in drivers/, 2 million LOC in arch/, and only 139 thousand LOC in kernel/. (http://unix.stackexchange.com/a/223753)

Edit: Would be nice to have the numbers for the latest Minix release for comparison. Does anybody know how big their core team is? (The kernel is something like 15k LOC IIRC.)

Manishearth · 9 years ago
Btw, if anyone is interested in learning how to write an OS, and interested in Rust, check out https://intermezzos.github.io/. It's a "learning OS" in Rust, with a companion book. Incomplete, but a good start.
iagooar · 9 years ago
Warning: I have no idea about systems programming.

I understand that the amount of lines of driver code comes from the variety of devices. But still, it looks completely unbalanced, when compared to the kernel code itself. So I have some questions here:

1) Shouldn't there be common interfaces / abstractions for most of the devices? 2) If they exist, could they be improved somehow? 3) A bit unrelated, but how fun / interesting is it to develop driver code?

pmlnr · 9 years ago
Actually, they can be really small... http://kolibrios.org/
jotux · 9 years ago
I worked at a company that made POS terminals on the OS development team. We had a proprietary OS based loosely on UNIX that was ~100KLOC (kernel). It was largely designed by 3-4 core developers and maintained by ~12 embedded software engineers (in addition to drivers, apps, UI, etc).
carussell · 9 years ago
> Does anybody know how big their core team is?

Right now it's about 2.

zongitsrinzler · 9 years ago
Are you the guy that says stuff like "How hard can it be?" and "It shouldn't take that long" in meetings?
notalaser · 9 years ago
Ours didn't have a GUI. It took us less than a year to write and the team was never larger than 6 people. It's less complicated than it seems at first sight.

There are a lot of things about machines that you need to understand, but the technology itself is relatively well-understood and sane practices are encouraged.

It blows my mind that less than twenty people can string up something non-trivial with Node.js et co.. Operating systems are trivial in comparison.

joepie91_ · 9 years ago
As somebody who develops significantly complex in Node.js and considers operating systems to largely be black magic, I'd love to have a chat about this some day, and compare notes :)
nickpsecurity · 9 years ago
Your website is interesting but a tad hard to follow due to the "organization" scheme. ;) So, what OS project did you work on in that first sentence and do you have a link to it?
nickpsecurity · 9 years ago
Wirth and a few others went to Xerox to see their personal computer with GUI that Steve Jobs also envied. They were told they couldn't buy it. "So, I built one..." (Wirth)

http://www.cfbsoftware.com/modula2/Lilith.pdf

Lilith was a custom computer with all kinds of hardware. The mouse eventually inspired Logitech. The software included a safe language (Modula-2), its compiler, an OS (Medos-2), a relational DB, and some other stuff. The trick was safe language, modularity, and simplicity in implementation details. Two or 3 people over a few years.

Later, that turned into the Oberon series of languages, compilers, and OS's. The extensions or ports usually took 1 or 2 students months to a year or two to pull off. Last one they released IIRC was A2 Bluebottle OS. Primitive, but usable. I keep telling people to rewrite it in other safe, C alternatives to save them work. Students got it done fast so I'm sure hobbyists could too.

https://github.com/Ignat99/a2

vram22 · 9 years ago
Interesting. I had read about Oberon (the OS) and probably Lilith in an old BYTE magazine issue years ago.
bane · 9 years ago
it's actually a pretty common task in undergrad CS programs for an individual or a small group (3-4 students) to write an *nix-like OS from the ground up, including most of the core utils, scheduler, process management, etc.

This is obviously much more full featured (includes a GUI), so this is actually a very nicely sized team for the task.

PaulRobinson · 9 years ago
You'd be surprised. There are lots of people writing operating systems as hobbies.

There's a wiki to get you started if you're interested: http://wiki.osdev.org/Main_Page

jakub_h · 9 years ago
Have you heard of Oberon? That's like three people in a year. Including building the computer. ;)
ape4 · 9 years ago
And rust only hit 1.0 on May 15, 2015 (I looked it up https://en.wikipedia.org/wiki/Rust_%28programming_language%2... )
jroesch · 9 years ago
Its actually much easier then it sounds, we had an undergrad write an entire windowing system on-top of implementing the rest of the OS for undergraduate operating systems at UW in a single quarter (2.5 months~).
jstimpfle · 9 years ago
Also relating to the whole thread, I think these kinds of statements are dangerous. What is a "windowing system" here? There is a huge spectrum. If a windowing system were so easy, why has Wayland taken such a long time? It's best to give a link to the project so people can see what problems were actually solved and how useful it is in practice.

I have been thinking hard about a simple plain text file format and associated tools for the last couple months. Sounds nowhere near as impressive. Yet I hope the end result could eventually make a practical difference.

Deleted Comment

kpozin · 9 years ago
I've been looking through some of the kernel code [1] out of curiosity, and I'm very surprised to see that most components have almost no internal documentation — minimal file and class comments, and even fewer inline comments. Is this typical of code for something as complex and central as an OS kernel? Are the developers planning to go back later and add documentation, or is the expectation that anyone who might need to work with this code will find its structure and details intuitive?

[1] https://github.com/redox-os/redox/tree/master/kernel/

tinco · 9 years ago
Yes, I like to read sources of big projects as a relaxing activity and almost never is documentation written in a way that a newbie can immediately catch up. The best projects will have a README.md in every directory that explains what a component or namespace is about, or nice big documentation headers. In my experience though if you really want to contribute the lack of documentation is only a small hurdle. If the code is well written then just following the code from the entrypoint on will give you a very good idea about what's going on. All the files and their names should make sense in under an hour or two.

In my opinion documentation is most important for external consumers of APIs who don't want to spend time grokking the code, and for explaining design decisions. For contributors it would be code quality.

sambeau · 9 years ago
There is this comment :) ..

    /// Into a cow
    pub fn into_cow<'a>(self) -> CowUrl<'a> {
        /*
             ______________
            < Mooooooooooo >
             --------------
                    \   ^__^
                     \  (oo)\_______
                        (__)\       )\/\
                            ||----w |
                            ||     ||
         */
        CowUrl::Owned(self)
    }

jaltekruse · 9 years ago
I only took a brief look over the code to try to find what documentation is actually there, but I managed to find this

https://github.com/redox-os/redox/blob/master/kernel/fs/url....

On your question of how typical a lack of system-internal documentation is in systems projects, the answer from my experience is unfortunately more common than it should be. There are a bunch of usage and design patterns that cannot be expressed regardless of the language you are using. I know very little about Rust, but I work mostly in Java, which is safer than many other languages (less flexible than interpreted or dynamic languages, less dangerous than C/C++). Even java, which has interfaces and strict typing can save you some of the more mundane unit tests and input validation code, has a bunch of design patterns on top of the core language. Projects could serve themselves well to document their code well to allow new contributors to look at any of the code and reasonably modify it to fix a bug or add a feature.

ivanceras · 9 years ago
In java, the jvm is the OS. Every object in Java is allocated in the heap, and doesn't provide developers direct control of a more optimized way of allocating objects. In rust, you can specify an object to be on the stack or on the heap, rust compiler is very smart and it knows where an object goes out of scope, thus not needed anymore and insert a code on that location to drop that object and free memory. Unlike in Java, which waits for the garbage collector to drop objects.

In a nutshell, java engineering effort is directed into the jvm(jit) to provide high performance when the applications are run. Rust engineering, on the other hand is directed on the compiler figuring out where and when an object is out of scope, no matter how complex it was attached to other objects, passed around multiple functions. It will figure it out.

kibwen · 9 years ago
There appears to be a book that serves as documentation, though I don't know how complete it is: http://www.redox-os.org/book/book/

Deleted Comment

yincrash · 9 years ago
I wonder why they decided to write their own coreutils instead of using https://github.com/uutils/coreutils
steveklabnik · 9 years ago
That project is attempting to re-create GNU-style coreutils, they wanted a slimmer, BSD style one, from what I understand.

Deleted Comment

progman · 9 years ago
Great news! Now it's worth to learn Rust just to keep up with this project.

Does Redox run in qemu only or also in VirtualBox? I tried it with all devices (USB, Network, etc.) disabled. Installation from ISO to /dev/sda worked fine but after reboot I got a hangup: "IDE Primary Master: Unknown Filesystem". Do I have to format the hardisk image with ZFS before installation?

progman · 9 years ago
Thanks for the recommendations!

First I have to install the current Rust version. Are there any MD5/SHA256/GPG to verify the integrity? Rust's download page doesn't provide anything like that.

https://www.rust-lang.org/downloads.html

steveklabnik · 9 years ago
http://static.rust-lang.org/dist/ has them. If you use multirust to install, and you have GPG installed, it will use them to check upon installation.
stryan · 9 years ago
qemu is recommended but Virtualbox is supported. Try running grabbing the latest source and running make virtualbox.
_0w8t · 9 years ago
So I can play with it on OSX ?
jszymborski · 9 years ago
The book has steps, but I think it runs through qemu
Nullabillity · 9 years ago
The installer worked under VirtualBox for me.
amelius · 9 years ago
I was reading this:

http://www.redox-os.org/book/book/design/urls_schemes_resour...

and I really got interested in the "everything is a URL" idea. But then I noticed that the most important parts of this text were missing :/

Perhaps somebody can clarify here.

Manishearth · 9 years ago
There's a bit more info here: https://github.com/redox-os/redox/wiki/URL
rch · 9 years ago
Interesting. They mention 9P, but their approach reminds me of 1060's NetKernel:

http://www.1060research.com/products/#roc

amelius · 9 years ago
Very interesting.

The only odd part is that the modules (drivers) themselves are not referenced by URL, but only by a simple word (in the example "port_io").

Also, I wonder how we could combine drivers. For example, (theoretically) instead of using "https" as driver, we could compose it as "HTTP over TLS over TCP", and change any of those subcomponents as desired. With URLs this might become clumsy.

Deleted Comment

Deleted Comment

tlrobinson · 9 years ago
That's pretty neat. You could kind of fake it on other OSes using something like Fuse with mount points like "/http" or "/tcp".
b3h3moth · 9 years ago
"Using Rust for an Undergraduate OS Course"[0] is an interesting reading about Rust, OS and C.

[0] http://rust-class.org/0/pages/using-rust-for-an-undergraduat...

fche · 9 years ago
How many "unsafe" declarations should one expect in regions of the code that will need to deal with untrusted data (like networks, syscalls)?
Manishearth · 9 years ago
I don't know about Redox, but Phil's Rust OS (http://os.phil-opp.com/) seems to manage to minimize unsafe Rust and only use it to create some low level OS abstractions that can be safely used. It's pretty neat.
bydo · 9 years ago
From the book[1]: A quick grep gives us some stats: The kernel has 16.52% unsafe code, a 50% improvement in the last three weeks. Userspace has roughly ~0.2%.

1: http://www.redox-os.org/book/book/introduction/unsafes.html

progman · 9 years ago
I guess not very much if Redox will support open hardware.

http://www.xda-developers.com/risc-v-cores-and-why-they-matt...

http://hackaday.com/2014/08/19/open-source-gpu-released/

http://www.openfpga.org

http://hackaday.com/2014/08/19/open-source-gpu-released/

http://open-ethernet.com

Some compelling projects (GPU) failed for lack of interest. The concepts behind those projects did not fail because most of the projects were isolated. The best time for open hardware is yet to come. Redox could accelerate that.

fche · 9 years ago
rust "unsafe" is not about proprietariness of the hardware. It's about doing lower level operations that, in order to express, the rust compiler's safety logic must be disabled.