The advantage (or disadvantage) of getting older: Seeing the younger generations come up with brilliant "new" ideas. New to them, whereas I think "yep, that was done 40 years ago, and 20 years ago, and now it's new all over again."
Part of the problem is the sheer breadth of IT and software development today. 40 years ago, I felt like I understood nearly everything: I designed processors at the transistor level, programmed from machine code through assembly to Lisp, Prolog, Pascal and other languages. Worked on projects from tiny to huge.
That's just not possible today. There are too many areas of application, too many frameworks and languages. Too many tools and platforms. Any individual developer is going to be in a niche. Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants.
Today, it's all you can do to stay abreast of developments in your own field. There is little time to look for lessons from the past, or across to ideas from other, parallel fields.
> Part of the problem is the sheer breadth of IT and software development today. 40 years ago, I felt like I understood nearly everything: I designed processors at the transistor level, programmed from machine code through assembly to Lisp, Prolog, Pascal and other languages. Worked on projects from tiny to huge.
The saddest thing about the way this changed is that the barrier to learning is so much lower today, but what do you actually do once you've learned?
When I was a kid, there were no programming classes in school and I had to go buy books and magazines to learn how to code in BASIC or assembler or Pascal. But once I've learned a new trick, I could immediately do something with it and feel good about it. It didn't matter it wasn't something spectacular, it didn't matter it wasn't very useful, because it was mine and there weren't 1000 of other things just like it, available to me at a touch of a button.
Compare that to what my kid is going through. He wanted to try learning to code, so we signed him up for some classes where he learned some Python. Later, in high school, he took AP CS Principles and they taught him some rudimentary JavaScript. Now, in his junior year, he's taking AP CS A, where they're teaching him Java.
Thing is, what can he do with all that? He's surrounded by devices running code. Writing a toy program that fills the screen with "HELLO, I AM COMPUTER" does not feel like you just learned a magical incantation that makes this machine do your bidding. Writing some actually useful code is harder, and it feels pointless, because there's almost always an existing app you can have at a click of a button and it does what you thought of and probably better. Writing useful code that doesn't already exist -- like e.g. your own mod for such-and-such game -- is even harder than that.
> Any individual developer is going to be in a niche. Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants.
To top off the irony, generalists are not being appreciated or sought much.
> Writing a toy program that fills the screen with "HELLO, I AM COMPUTER" does not feel like you just learned a magical incantation that makes this machine do your bidding.
But there's joy to be found in writing pointless and abstract programs. It's more about the journey and less about the utility of the finished product. Writing, for example, an ASCII-art drawing system that works on a big rectangular 2D character array can teach you so many concepts – from OO and encapsulation, through linear algebra, to file IO.
> Writing a toy program that fills the screen with "HELLO, I AM COMPUTER" does not feel like you just learned a magical incantation that makes this machine do your bidding.
No, but in the 80s, when the guy at Radio Shack didn't know how to stop it, it was still worth it somehow.
What can your kid do to assert some goofy kind of power in today's adult world?
> Writing some actually useful code is harder, and it feels pointless, because there's almost always an existing app you can have at a click of a button and it does what you thought of and probably better.
Well, there are a lot of people who enjoy cooking, or gardening, or carpentry, or making their own clothing, or playing in a local softball league, even though there are many professionals with professional-level skills and equipment who can do those jobs better than they can.
I think it's like you suggested in your later comment: you either have the bug for programming, or you don't. There are a lot of programmers nowadays who don't -- they just program to make a buck.
Not that there's anything wrong with making a buck, but it's like the guy who flips burgers at McDonalds. He does it to make a buck, not because it's speaking to his inner need to be a creative chef.
> Writing a toy program that fills the screen with "HELLO, I AM COMPUTER" does not feel like you just learned a magical incantation that makes this machine do your bidding
Find out what his favourite hobby or favourite game is, then demonstrate a little script... Suddenly the conditional branches and loops don't seem so abstract anymore !
Isn't there is a similar spread for older technologies and crafts like horticulture/agriculture, textiles, ceramics, metalwork, printing, music, cooking... is this just evidence that software has arrived?
>That's just not possible today. There are too many areas of application, too many frameworks and languages. Too many tools and platforms.
I find a lot of these tools and frameworks and languages and platforms are attempts to correct perceived problems in the original tools and methods. But these new things have their own issues which someone else will attempt to correct. Then you get your cliques and reddit subs that make fun of you for using X because it's old.
My point is that there is no back to basics--no grounding in the fundamentals we grew up with--or should have at least.
I think you do not understand just how much stuff it out there. Just in the web world, can you do React and Angular and NodeJS and Spring? Have you really looked at the CSS specification? That's a career in itself.
Moving slightly to mobile apps, what about Jetpack Compose for Android and the equivalent for iPhones? Electron? Or classic programming: How is your SQL, including stored procedures? Note that Oracle is different from Microsoft is different from MySQL. Want to learn ABAP? How about writing a back-end system in Java, with a front-end in JavaFX?
I'm just getting started...
No, you cannot learn all of this sufficiently to be even remotely productive. You may, like me, be aware of a lot of it. I'm sure you could pick any single new platform or technology, and become productive in it in a few weeks or months. But to be a real generalist, able to do it all? That is simply no longer possible.
> Part of the problem is the sheer breadth of IT and software development today
No, the problem is wholly human limitation. All humans start as an ignorant babe and have to both mentally develop and learn everything from scratch. This includes the rebellious teenage years and not listening to the wisdom of their elders (and for the adults around them, how to properly raise and mentor kids).
We have been doing this exact dance every dozen years for thousands of years for every single field and every single little task, from how to cook the best bread, the best way to saddle an ox, how to fix a toilet, etc. Learning and discovering, and relearning and rediscovering, ad infinitum.
This is a common thing I've read in books and heard from my more experience co workers. My favorite current example is OpenAPI specs for building a client or generating docs. SOAP was another big thing. So was CORBA. We're really good at rebuilding RPCs. Not to say that they don't have their differences and benefits.
NextJS always gives me a laugh too. I remember when it was newer in 2019/2020 and people raving about its file based routing and server-side rendering.
These new waves sometimes have new benefits to them, and sometimes they don't. I have to imagine that we won't ever seen an end to it, but at least it keeps some exciting new stuff on the table.
Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants.
40 years ago: your physicist knows nothing of how to write RPG, your mainframe developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin wouldn't touch FORTRAN with a ten-foot pole.
... Which is to say, I don't really buy it. Or rather, I don't think that specialization is a recent phenomenon. It is true that the personal computing market consisted of comparatively simpler machines, which allowed for greater breadth of understanding.
We call this reinventing the flat tire. In the old days, you would chastise people for reinventing the wheel. Now we beg, “Oh, please, please reinvent the wheel.” -- Alan Kay
On the contrary, the widespread obsession with "Unix philosophy" often provides quasi-religious objections to suggestions we ever try new approaches to things. In particular, the desire to cram increasingly diverse and rich interfaces into 70s file IO is a strange sacrament
That approach works so well that a program written in '70s is so well abstracted away from the intricacies of the system that it can input data from a network socket, a pipe, a file, or output to whatnot, and doesn't need to be touched for all that to work. A plain program reading from stdin and writing to stdout can be turned into a network service via inetd or similar by just redirecting file descriptors.
But for the contemporary software coders the ingenious idea is of course rubbish. They would replace everything by node_modules.
It's great that it is easy to do simple things with ancient tools, and it's completely true that you can inetd and CGI your way to covering numerous use cases on small systems.
The problem is the lack of innovation holds us back from solving complex use cases. The sockets API in particular is a gigantic mess, because the protocols are more complicated than people imagined forty years ago. For example it is simple to send a datagram. What if you want to get notified of the timestamp when the datagram hits the wire? Now you will 1) read a lot of man pages, 2) write 100s of lines of gross C code, 3) read the source code of the Linux kernel to figure out which parts of the manual were lies, then 4) fix your program with some different code and a long, angry comment. None of they would be necessary if the last quarter-century of network features hadn't been jammed into the ioctl and setsockopt hack interfaces.
In other words there's plenty of room for operating systems to reconsider interfaces in light of the systems and protocols we have now, instead of Space Age imaginary use cases.
The devil is in the details. You can't turn really a command from the 70s into a webservice because its input are command line options/arguments and not query params or RPC arglists, and its output is probably intended for humans to read or parse easily with a regex/sed/awk. Web services speak in structured data for their input and output over protocols that can't be abstracted as file i/o.
Because they're not just files. There are additional semantics and concerns that you cannot abstract away with read/write calls. You also need ioctls, you need send/recv, you need parsers on top to do something useful with the data, and so on.
The reason that the simple model dies out is because it's not as useful as it seems in practice.
It was originally developed as a multi-user time-sharing system. Multiple users would access the system simultaneously via the attached teletype terminals. The "shell" program handled such an interactive user session. All IO was blocking, and programs were written in a sequential manner. Piping enabled them to feed the standard output of one process to the standard input of another.
It's a good framework for text-based computing, but nowadays we have so much more problems to cope with. This original idea never addressed networking, binary versioning and control, configuration management, async IO, multithreading, GUI, etc.
You're not inventing new things, take a look at CORBA and compare that to COM objects. It's not "no new things", it's "build things that are actually principled, don't just cram a bunch of garbage together because it's the path of least resistance".
>Every increase in expressiveness brings an increased burden on all who care to understand the message.
There are plenty of new things that are largely compatible with unix philosophy. MQTT is one of my favorite examples, it's a message bus that follows a lot of unix philosophy and I find it a joy to work with compared to stuff like DBUS. Obviously it doesn't fill the same role as DBUS, but still.
You can't blame Unix for getting popular, or for almost no research and progress in operating systems since Unix got popular. Rob Pike wrote a good paper about that.
At this point it's network effect, familiarity, and the free price tag, not a quasi-religious sacrament. Anyone can try to take that different approach, but that doesn't mean it will go anywhere. Some very smart people have worked on alternatives to Unix, or different directions for operating systems -- including the original Unix team with Plan 9, Wirth with Oberon, the Lisp machines. No Unix Inquisition shut those down, they just didn't offer a 10x improvement along enough axes.
What are some alternatives that you'd recommend people look at?
I'm keen for any potential successor, but a lot things come back to effectively text, and anything we could build on top of it. Binary object formats seem like an alternative for faster parsing of structured data, so while that ability has always been it, it's a matter of people actually sticking to it. Maybe some coordination between the program and the shell?
Some alternatives is a structured data interface, a bit like PowerShell where not everything needs to be serialized and deserialized and parsed in every program in a pipeline.
Another approach is that the OS is basically a complete environment where everything is code. You can see the idea in the Smalltalk environments where you can theoretically interact with every object by sending messages. Lisp machines come to mind as well and one could even consider early personal computers that booted into BASIC as an idea of this (though in BASIC instead ov everything is a file everything is memory)
One alternative is Apache Arrow. It's an efficient in-memory format used to represent flat or hierarchal data. Multiple databases and programs like pandas support this, so they become compatible with each other.
> What are some alternatives that you'd recommend people look at?
For me, this would mean an OS that supports both static and runtime reflection, and therefore code-generation and type generation. Strong typing should become a fundamental part of OS and system design. This probably means having an OS-wide thin runtime (something like .NET?) that all programs are written against.
UNIX's composability was a good idea, but is an outdated implementation, which is built on string-typing everything, including command-line options, environment variables, program input and output, and even memory and data itself ('a file is a bag of bytes').
The same thing has happened to C (and therefore C++) itself. The way to use a library in C (and C++) is to literally textually include function and type declarations, and then compile disparate translation units together.
If we want runtime library loading, we have to faff with dlopen/dlsym/LoadLibrary/GetProcAddress, and we have to know the names of symbols beforehand. A better implementation would use full-fledged reflection to import new types, methods, and free functions into the namespace of the currently-executing program.
James Mickens in The Night Watch[1] says 'you can’t just place
a LISP book on top of an x86 chip and hope that the hardware
learns about lambda calculus by osmosis'. Fair point. But the next-best thing would have been to define useful types around pointers as close to the hardware as possible, rather than just saying 'here, this is the address for the memory you want from HeapAlloc; go do what you want with it and I will only complain at runtime when you break it'. Pointers are a badly leaky abstraction that don't really map to the real thing anyway, given MMUs, memory mapping, etc.
There are so many ways we could make things a bit more disciplined in system and OS design, and drastically improve life for everyone using it.
Of the major vendors, I'd say only Windows has taken merely half-hearted steps towards an OS-wide 'type system'. I say half-hearted, because PowerShell is fantastic and handles are significantly more user-friendly than UNIX's file descriptors, thread pointers, and pipes. But this is still a small oasis in a desert of native programs that have to mess with string-typing.
> In particular, the desire to cram increasingly diverse and rich interfaces into 70s file IO is a strange sacrament
Long time ago I asked my karate sensei how come I see some of the black belts doing katas in a different way. His answer was that to improve on something one must first have deep mastery of how it is done.
(i.e. "Shut up and do the katas correctly. If you ever get to black belt then you can improvise.")
Always found that answer to apply to everything. In particular it applies to a lot of the poor quality libraries and code we see today as a direct consequence of not understanding the past and what has been tried and failed and why.
The fact that we're talking in 2024 about using the file abstractions of Unix created in the 70s is proof of how extremely powerful they are. Nothing is perfect, but that's quite an achievement.
A mistake we often make I believe is to focus on code-quality, rather than code-content-quality. Code is basically mathematics. Is it good math or bad that is the question. Does it do something useful, in an economical way.
To write great code, I believe, means you have to be a domain expert of the domain for which your code solves problems.
"Everything is a file" is not a lesson, it is an imposition that with some some things that are very different to files it is a very wrong thing.
The lesson is: Minimise as much as possible the different interfaces necessary and reuse them. Interlisp or Lisp machines used Expressions so everything could be read and written by everything.
Unix is a mess of binary formats, a mess of different console emulator formats, a mess of different text formats.
And the article is just about one guy who does not like Wayland not using files for everything, which is fine for me.
The main lesson from unix that is getting lost is composability of systems.
So many silos and proprietary "cloud" systems, so much glue that needs to be constantly rewritten because there isn't any stability in an ecosystem composed of competing rather than cooperating actors.
There are great things that have come along, I would say k8s has been very successful at introducing a highly interoperable, composable API for abstracting over a bunch of machines.
But the trend towards hosted services that are incentivized to create lock-in and data-gravity isn't great.
I don't think this lesson is getting lost accidentally, but intentionally. There was a brief moment in the late 00's/early 10's where companies were creating these rich web API's that would let you (and other services acting on your behalf) interact with their system in arbitrary and novel ways. But that turned out to not be very profitable (or at the very least, wouldn't let you create a walled garden that would get 100x exit for your VC's) so we've steadily moved to more locked down, less interactive, less open, less useful, but ultimately more profitable services.
It's not just unix, but programming languages themselves and general patterns for how software is built and how projects are run. It's what I'm most frustrated by in this industry. So much waste. I once thought surely this industry would mature, but now I don't see that happening in my lifetime. Reminds me of human civilizations before people developed language and wrote things down and almost all knowledge was lost with each new generation, keeping progress limited to a snails pace.
It’s a ramble that ends up with a complaint that Wayland doesn’t fit into the stream file model that’s supposedly critical to “unix”. Block files have been in Unix forever and they don’t either.
Maybe read “Worse is Better” before writing part 3.
And “Unix” is orthogonal with FLOSS.
Sorry to be such a downer — been reading el reg since it started.
I read the article, and I didn't see this point made. Mostly it was a very long description of the history of Unix, with a tangent about microkernels.
Your headings are
What is 'Unix' anyway?
Generations and the gaps between
Turn it up to 11
I believe your ultimate point is about Wayland and perhaps that it should follow everything-is-a-file. But that comes so late in the article that you barely give it a passing mention, and it isn't consistent with the premise of the article.
Nah. :-) I am not terribly interested in Wayland, myself, and I suspect that in the fullness of time it will turn out to be a not-very-interesting blind alley.
I linked to it in the first paragraph. You can read the script if you want.
Redox, OTOH, no: it does not even enter into it. Although I do not address it, if pressed, I might concede that if there is a Rust involvement, it's R9, the successor to Harvey OS.
I'm really not a fan of UNIX in the slightest, other than as a basic kernel OS to build something nicer on, like what macOS did. I think it's much better to make it easier to write programs and have a rich system library than to glue together text streams. And in fact that's basically what has happened instead: the "UNIX philosophy" has largely been replaced by Python and the like. People now reach for Python for many tasks that were previously shell one-liners, and imo that's a positive development.
When I was a kid learning how to code reading online, everyone was religiously defending the "UNIX philosophy" showing off their fancy and indecipherable one-liners (that don't work on my system bc I have GNU awk instead of whatever awk they have or whatever). This has basically died out year by year as so many people switch over to scripting languages with great standard libraries and super-powered easy to install libraries.
Now, I'm not really a fan of the super overkill object-oriented approach that systems like mac and smalltalk take, but I think our modern approach is a good compromise
On the topic of "UNIX shell one-liners versus Python", I enjoy letting my students solve the "make a list of the words in a file with their counts" exercise in Python, then I show them a sh one-liner.
The Python scripts takes them from 10 minutes to two hours, they're buggy, and they cannot process big files - everything is in RAM (with a monster dict) - without a crash/system freeze.
The one thing holding Python back from completely eating my workflow of utils, glue, and hacks is environment isolation. Nothing sucks more than when activating some venv breaks a background tool I wrote 3 years ago and forgot about. I recognize that this is a surmountable problem with at most moderate effort, but getting to a point of complete frictionlessness would really move the whole ecosystem forward a lot, IMO.
The random Wayland bashing seems both completely unrelated to whatever the topic of the article is, and uncalled for. Wayland literally is a protocol that is communicated over a file (WAYLAND_DISPLAY). Sure, the file is a (UDS) socket, but it is still a file! Also Wayland is purely an userland thing, so it is very much constrained on what primitives kernel has to offer.
The way the Wayland section is included there, it seems like author is heavily implying that it is prime example of "spiralling software complexity and bloat", which to me sounds mostly like they do not know what they are talking about. Core Wayland is distinctly smaller and simpler than X11 ever was. Sure, modern graphics (etc) are intrinsically complex and Wayland exposes some that, but it does it mostly by keeping Wayland itself as simple and transparent as possible, instead of making Wayland (or rather its implementations) hugely complex and bloated.
But besides this Wayland thing, I'm really confused what the article is actually trying to say and how does the title relate in any way to the content? The intro implies that there are some lessons to be learned from UNIX history, but the article sure as hell doesn't cast any light on what those lessons might be. There is also this implication that we are somehow coding ourselves into a corner, but that is never actually expanded on what they are meaning with that?
In terms of lessons learned, the key observation maybe is that UNIX peaked somewhere around SysIII before BSD etc hippies set their grubby hands on it and messed everything up.
Readit News
Part of the problem is the sheer breadth of IT and software development today. 40 years ago, I felt like I understood nearly everything: I designed processors at the transistor level, programmed from machine code through assembly to Lisp, Prolog, Pascal and other languages. Worked on projects from tiny to huge.
That's just not possible today. There are too many areas of application, too many frameworks and languages. Too many tools and platforms. Any individual developer is going to be in a niche. Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants.
Today, it's all you can do to stay abreast of developments in your own field. There is little time to look for lessons from the past, or across to ideas from other, parallel fields.
The saddest thing about the way this changed is that the barrier to learning is so much lower today, but what do you actually do once you've learned?
When I was a kid, there were no programming classes in school and I had to go buy books and magazines to learn how to code in BASIC or assembler or Pascal. But once I've learned a new trick, I could immediately do something with it and feel good about it. It didn't matter it wasn't something spectacular, it didn't matter it wasn't very useful, because it was mine and there weren't 1000 of other things just like it, available to me at a touch of a button.
Compare that to what my kid is going through. He wanted to try learning to code, so we signed him up for some classes where he learned some Python. Later, in high school, he took AP CS Principles and they taught him some rudimentary JavaScript. Now, in his junior year, he's taking AP CS A, where they're teaching him Java.
Thing is, what can he do with all that? He's surrounded by devices running code. Writing a toy program that fills the screen with "HELLO, I AM COMPUTER" does not feel like you just learned a magical incantation that makes this machine do your bidding. Writing some actually useful code is harder, and it feels pointless, because there's almost always an existing app you can have at a click of a button and it does what you thought of and probably better. Writing useful code that doesn't already exist -- like e.g. your own mod for such-and-such game -- is even harder than that.
> Any individual developer is going to be in a niche. Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants.
To top off the irony, generalists are not being appreciated or sought much.
But there's joy to be found in writing pointless and abstract programs. It's more about the journey and less about the utility of the finished product. Writing, for example, an ASCII-art drawing system that works on a big rectangular 2D character array can teach you so many concepts – from OO and encapsulation, through linear algebra, to file IO.
No, but in the 80s, when the guy at Radio Shack didn't know how to stop it, it was still worth it somehow.
What can your kid do to assert some goofy kind of power in today's adult world?
Well, there are a lot of people who enjoy cooking, or gardening, or carpentry, or making their own clothing, or playing in a local softball league, even though there are many professionals with professional-level skills and equipment who can do those jobs better than they can.
I think it's like you suggested in your later comment: you either have the bug for programming, or you don't. There are a lot of programmers nowadays who don't -- they just program to make a buck.
Not that there's anything wrong with making a buck, but it's like the guy who flips burgers at McDonalds. He does it to make a buck, not because it's speaking to his inner need to be a creative chef.
Find out what his favourite hobby or favourite game is, then demonstrate a little script... Suddenly the conditional branches and loops don't seem so abstract anymore !
What has been will be again,
what has been done will be done again;
there is nothing new under the sun.
10
Is there anything of which one can say,
“Look! This is something new”?
It was here already, long ago;
it was here before our time.
11
No one remembers the former generations,
and even those yet to come
will not be remembered
by those who follow them.
---
That's from Ecclesiastes 1 (NIV), and it was written at some point between 450BC and 180BC.
I find a lot of these tools and frameworks and languages and platforms are attempts to correct perceived problems in the original tools and methods. But these new things have their own issues which someone else will attempt to correct. Then you get your cliques and reddit subs that make fun of you for using X because it's old.
My point is that there is no back to basics--no grounding in the fundamentals we grew up with--or should have at least.
Moving slightly to mobile apps, what about Jetpack Compose for Android and the equivalent for iPhones? Electron? Or classic programming: How is your SQL, including stored procedures? Note that Oracle is different from Microsoft is different from MySQL. Want to learn ABAP? How about writing a back-end system in Java, with a front-end in JavaFX?
I'm just getting started...
No, you cannot learn all of this sufficiently to be even remotely productive. You may, like me, be aware of a lot of it. I'm sure you could pick any single new platform or technology, and become productive in it in a few weeks or months. But to be a real generalist, able to do it all? That is simply no longer possible.
No, the problem is wholly human limitation. All humans start as an ignorant babe and have to both mentally develop and learn everything from scratch. This includes the rebellious teenage years and not listening to the wisdom of their elders (and for the adults around them, how to properly raise and mentor kids).
We have been doing this exact dance every dozen years for thousands of years for every single field and every single little task, from how to cook the best bread, the best way to saddle an ox, how to fix a toilet, etc. Learning and discovering, and relearning and rediscovering, ad infinitum.
NextJS always gives me a laugh too. I remember when it was newer in 2019/2020 and people raving about its file based routing and server-side rendering.
These new waves sometimes have new benefits to them, and sometimes they don't. I have to imagine that we won't ever seen an end to it, but at least it keeps some exciting new stuff on the table.
40 years ago: your physicist knows nothing of how to write RPG, your mainframe developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin wouldn't touch FORTRAN with a ten-foot pole.
... Which is to say, I don't really buy it. Or rather, I don't think that specialization is a recent phenomenon. It is true that the personal computing market consisted of comparatively simpler machines, which allowed for greater breadth of understanding.
But for the contemporary software coders the ingenious idea is of course rubbish. They would replace everything by node_modules.
The problem is the lack of innovation holds us back from solving complex use cases. The sockets API in particular is a gigantic mess, because the protocols are more complicated than people imagined forty years ago. For example it is simple to send a datagram. What if you want to get notified of the timestamp when the datagram hits the wire? Now you will 1) read a lot of man pages, 2) write 100s of lines of gross C code, 3) read the source code of the Linux kernel to figure out which parts of the manual were lies, then 4) fix your program with some different code and a long, angry comment. None of they would be necessary if the last quarter-century of network features hadn't been jammed into the ioctl and setsockopt hack interfaces.
In other words there's plenty of room for operating systems to reconsider interfaces in light of the systems and protocols we have now, instead of Space Age imaginary use cases.
Because they're not just files. There are additional semantics and concerns that you cannot abstract away with read/write calls. You also need ioctls, you need send/recv, you need parsers on top to do something useful with the data, and so on.
The reason that the simple model dies out is because it's not as useful as it seems in practice.
It's a good framework for text-based computing, but nowadays we have so much more problems to cope with. This original idea never addressed networking, binary versioning and control, configuration management, async IO, multithreading, GUI, etc.
>Every increase in expressiveness brings an increased burden on all who care to understand the message.
There are plenty of new things that are largely compatible with unix philosophy. MQTT is one of my favorite examples, it's a message bus that follows a lot of unix philosophy and I find it a joy to work with compared to stuff like DBUS. Obviously it doesn't fill the same role as DBUS, but still.
http://doc.cat-v.org/bell_labs/utah2000/
At this point it's network effect, familiarity, and the free price tag, not a quasi-religious sacrament. Anyone can try to take that different approach, but that doesn't mean it will go anywhere. Some very smart people have worked on alternatives to Unix, or different directions for operating systems -- including the original Unix team with Plan 9, Wirth with Oberon, the Lisp machines. No Unix Inquisition shut those down, they just didn't offer a 10x improvement along enough axes.
I'm keen for any potential successor, but a lot things come back to effectively text, and anything we could build on top of it. Binary object formats seem like an alternative for faster parsing of structured data, so while that ability has always been it, it's a matter of people actually sticking to it. Maybe some coordination between the program and the shell?
Another approach is that the OS is basically a complete environment where everything is code. You can see the idea in the Smalltalk environments where you can theoretically interact with every object by sending messages. Lisp machines come to mind as well and one could even consider early personal computers that booted into BASIC as an idea of this (though in BASIC instead ov everything is a file everything is memory)
For me, this would mean an OS that supports both static and runtime reflection, and therefore code-generation and type generation. Strong typing should become a fundamental part of OS and system design. This probably means having an OS-wide thin runtime (something like .NET?) that all programs are written against.
UNIX's composability was a good idea, but is an outdated implementation, which is built on string-typing everything, including command-line options, environment variables, program input and output, and even memory and data itself ('a file is a bag of bytes').
The same thing has happened to C (and therefore C++) itself. The way to use a library in C (and C++) is to literally textually include function and type declarations, and then compile disparate translation units together.
If we want runtime library loading, we have to faff with dlopen/dlsym/LoadLibrary/GetProcAddress, and we have to know the names of symbols beforehand. A better implementation would use full-fledged reflection to import new types, methods, and free functions into the namespace of the currently-executing program.
James Mickens in The Night Watch[1] says 'you can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis'. Fair point. But the next-best thing would have been to define useful types around pointers as close to the hardware as possible, rather than just saying 'here, this is the address for the memory you want from HeapAlloc; go do what you want with it and I will only complain at runtime when you break it'. Pointers are a badly leaky abstraction that don't really map to the real thing anyway, given MMUs, memory mapping, etc.
There are so many ways we could make things a bit more disciplined in system and OS design, and drastically improve life for everyone using it.
Of the major vendors, I'd say only Windows has taken merely half-hearted steps towards an OS-wide 'type system'. I say half-hearted, because PowerShell is fantastic and handles are significantly more user-friendly than UNIX's file descriptors, thread pointers, and pipes. But this is still a small oasis in a desert of native programs that have to mess with string-typing.
[1]: https://www.usenix.org/system/files/1311_05-08_mickens.pdf
Long time ago I asked my karate sensei how come I see some of the black belts doing katas in a different way. His answer was that to improve on something one must first have deep mastery of how it is done.
(i.e. "Shut up and do the katas correctly. If you ever get to black belt then you can improvise.")
Always found that answer to apply to everything. In particular it applies to a lot of the poor quality libraries and code we see today as a direct consequence of not understanding the past and what has been tried and failed and why.
The fact that we're talking in 2024 about using the file abstractions of Unix created in the 70s is proof of how extremely powerful they are. Nothing is perfect, but that's quite an achievement.
To write great code, I believe, means you have to be a domain expert of the domain for which your code solves problems.
https://www.theregister.com/2023/12/25/the_war_of_the_workst...
Note that this is loosely based on part of a FOSDEM talk I gave 4 years ago.
If you impose any structure, you're telling hackers what to do at the system level.
If that clashes with their favorite language run-time or whatever, you're effectively declaring war.
"Everything is a file" is not a lesson, it is an imposition that with some some things that are very different to files it is a very wrong thing.
The lesson is: Minimise as much as possible the different interfaces necessary and reuse them. Interlisp or Lisp machines used Expressions so everything could be read and written by everything.
Unix is a mess of binary formats, a mess of different console emulator formats, a mess of different text formats.
And the article is just about one guy who does not like Wayland not using files for everything, which is fine for me.
Very very strongly agree. As a corollary a small number of too simple interfaces is better than a larger number of more sophisticate interfaces.
"It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures."
- Alan Perlis' Epigrams on Programming (1982).
So many silos and proprietary "cloud" systems, so much glue that needs to be constantly rewritten because there isn't any stability in an ecosystem composed of competing rather than cooperating actors.
There are great things that have come along, I would say k8s has been very successful at introducing a highly interoperable, composable API for abstracting over a bunch of machines.
But the trend towards hosted services that are incentivized to create lock-in and data-gravity isn't great.
Emblematic of this journey is Twitter, I think.
This is by me on El Reg, and it's part 2 of this year's FOSDEM talk. It will (I hope!) make more sense in context with the rest.
Maybe read “Worse is Better” before writing part 3.
And “Unix” is orthogonal with FLOSS.
Sorry to be such a downer — been reading el reg since it started.
Way ahead o' ya, kid.
https://www.theregister.com/2023/12/25/the_war_of_the_workst...
Note that this is loosely based on part of a FOSDEM talk I gave 4 years ago.
Your headings are
What is 'Unix' anyway?
Generations and the gaps between
Turn it up to 11
I believe your ultimate point is about Wayland and perhaps that it should follow everything-is-a-file. But that comes so late in the article that you barely give it a passing mention, and it isn't consistent with the premise of the article.
Nah. :-) I am not terribly interested in Wayland, myself, and I suspect that in the fullness of time it will turn out to be a not-very-interesting blind alley.
The talk was a couple of weeks ago now and is already online: https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-...
I linked to it in the first paragraph. You can read the script if you want.
Redox, OTOH, no: it does not even enter into it. Although I do not address it, if pressed, I might concede that if there is a Rust involvement, it's R9, the successor to Harvey OS.
https://github.com/r9os/r9
Thanks!
I had some great feedback from some 9Front folks and ended up meeting up with a bunch of them after the event.
Given that I was half-expecting a lynch mob, this was a very pleasant surprise indeed... ;-)
Deleted Comment
When I was a kid learning how to code reading online, everyone was religiously defending the "UNIX philosophy" showing off their fancy and indecipherable one-liners (that don't work on my system bc I have GNU awk instead of whatever awk they have or whatever). This has basically died out year by year as so many people switch over to scripting languages with great standard libraries and super-powered easy to install libraries.
Now, I'm not really a fan of the super overkill object-oriented approach that systems like mac and smalltalk take, but I think our modern approach is a good compromise
The Python scripts takes them from 10 minutes to two hours, they're buggy, and they cannot process big files - everything is in RAM (with a monster dict) - without a crash/system freeze.
The power of UNIX, pipes and re-use.
How many words are there in English? 1M? Huge text [Unicode] files.
Fancier tokenizer:
The way the Wayland section is included there, it seems like author is heavily implying that it is prime example of "spiralling software complexity and bloat", which to me sounds mostly like they do not know what they are talking about. Core Wayland is distinctly smaller and simpler than X11 ever was. Sure, modern graphics (etc) are intrinsically complex and Wayland exposes some that, but it does it mostly by keeping Wayland itself as simple and transparent as possible, instead of making Wayland (or rather its implementations) hugely complex and bloated.
But besides this Wayland thing, I'm really confused what the article is actually trying to say and how does the title relate in any way to the content? The intro implies that there are some lessons to be learned from UNIX history, but the article sure as hell doesn't cast any light on what those lessons might be. There is also this implication that we are somehow coding ourselves into a corner, but that is never actually expanded on what they are meaning with that?
In terms of lessons learned, the key observation maybe is that UNIX peaked somewhere around SysIII before BSD etc hippies set their grubby hands on it and messed everything up.