Like the author, I am saddened by systemd. I'm not rabidly opposed to it. I use it because Debian uses it and I like debian. And in some ways, I like the consistency better than the plethora of init script/run levels I used to have to deal with. But it does lack (to me) the Unix gestalt of having composable little pieces that could be pretty well put together and are each individually documentable and compose well in conceptual space as well. There was less surprises and nuanced side effects.
As an init manager, systemd is the best thing that has happened to the wider linux ecosystem. Being able to indicate dependencies, document order and being able to let an application tell the init manager it is done and dependents of it can be started makes starting up way better.
I understand the downsides people have of systemd, but I have the feeling the huge upside is often overlooked.
I've worked with init.d style init systems that had those features using special comments and sourced helper functions, but I bet if you wanted to do it all properly you'd end up with something like systemd. Or GNU Shepherd!
It is more than merely an "init manager". And I disagree that it is the best thing ever - it is perfectly possible to operate linux without systemd.
> but I have the feeling the huge upside is often overlooked.
It is fine to objectively compare trade-offs. However had, it has to be a fair comparison; we can not start with "init manager" because systemd does a lot more, so how can a comparison to any software with less code be fair then? runit doesn't do much more than for initializing.
I wholeheartedly I agree with the author and you. It is sad how Linux is incrementally reshaped based on ideas which often betray the original principles that made UNIX, it derivatives, and copies great. But it is on us to create alternatives. It is free software after all.
Principles that Aix, HP-UX, Solaris, DG/UX, Tru64, NeXTSTEP,.... have forgotten long before FOSS circles keep repeating them as some ideology kind of thing.
Great timing this article. I came to HN to escape a frustration point where this blooming script behaves differently when wrapped in a systemd service than when I run it on its own. I suspect there's some systemd punctuatory dance with a throwaway character here or there to semaphore me off to neverneverland. It's so inscrutable sometimes.
If someone is using Debian it’s likely because they want no frills Linux, with apt-get, and the reliable cadence of LTS releases. And the added benefit of if they run into a problem it’s likely somebody else has as well
As a former hater of systemd it surprises me that people still have this disdain of systemd especially due to the so called "Unix philosophy", which in the Linux world mainly survives due to GNU writing GNUtils a rewriting of the Unix commands.
I remember embracing systemd funny enough due to when I was running alpine as my server and I had to write my own r script and boy did that quickly make me remember the awful times on freebsd debugging sh.
Same for me, I remember trying to configure cron to run a task periodically and it just did not budge. No error but also did not work. Systemd was working on first attempt with basic debugging when timer will be triggered again and what is the log of executed process after it has been triggered. Also manually triggering something to test if command is correct? Awesome.
This is precisely why it is valid to say that systemd doesn't align with the Unix philosophy, and yet it's also valid that many people such as yourself like it.
I don't want my init system to replace cron. I don't want it to manage logging. I don't want it to have any debugging capabilities. All of these things can be done with other programs, arguably in a much more flexible and robust way.
An init system should do one thing (well): manage system services. Within that context, it should start services on boot, keep them running in the background, and allow the user to create, stop, and start services. That's it. And it could be argued that even those responsibilities are too broad for a single program.
So I understand that you and many others like that systemd provides solutions to all of these tasks in an integrated way. But you should also understand that this does in fact go against the Unix philosophy, of small, independent, but composable programs with a single responsibility. It "proposes" alternatives to many other tools for no particular reason, until users are effectively using GNU/systemd/Linux.
And, yes, I know that technically systemd is not a monolith and is composed of many programs, but that is a moot point. It is a single project, maintained by a strongly opinionated team, and given the importance of it, most distros go all-in, so users are strongly recommended, if not forced, to use all of its programs. In many cases it's not even possible to use one individual systemd program idependently from another. This is why systemd is seen as a kraken that takes over the entire system if you do decide to use one part of it.
Forget Unix philosophy. The Linux philosophy suggests the monolithic kernel and, therefore, monolithic userland. Systemd fits perfectly, was pushed more or less openly by Linus himself and won. In Linux we don't want a million scripts but large binaries.
This shows a lack of understanding. I'll skip repeating the issues with regard to systemd here, so instead on just a few points you mentioned.
- You claim that the Unix philosophy only survives to the GNUtils. Well, that shows to me a lack of understanding what the philosophy is about. Everything is a file is similar to the OOP approach of everything is an object. I recommend watching Ken Thompson when he was young here: https://www.youtube.com/watch?v=tc4ROCJYbm0
It does not capture all the UNIX philosophy but it does extend on the reasons why that philosophy works well. The philosophy is bigger than that of course but it helps serve as a counter-argument.
- The example of "writing your own script" is no different to a non-systemd system. Why would a script work or not work based on systemd? You mention as example FreeBSD debugging a shell script. Well, others use proper languages such as ruby or python. Everything that can be done via systemd I can do without it too and, in fact, have been doing so. Ruby essentially runs my system as the primary layer on everything (granted, it runs Linux, and thus mostly C, and ruby is at the end of the day a syntactic wrapper over C). I never understood why systemd would matter. I read the advertisement of the systemd devs - none of this applied to my use cases, so I never "embraced" systemd, simply because I never needed it. I did point out the increased complexity of it as a negative trade-off and this has been true til this day.
- Former "hater" also implies that criticism is not based on rationale and logic. This is not the case either. It's funny to me how the pro-systemd camp isn't really able to come up with compelling arguments on their own.
I would say your comment either shows a lack of understanding, or that you completely missed the point.
> The example of "writing your own script" is no different to a non-systemd system. Why would a script work or not work based on systemd?
Of course, you can write the service itself in Python or Ruby or whatever regardless the service manager. The point is that with systemd, or upstart, or other service managers like this that make things more declarative (launchd?), you don't have to write a script to manage the service at all.
On systemd, you declare which services yours depend on, how to run it, which user should be used to run your service, and many things are handled for you, including many security mechanisms you don't need to think about and provide further config for this stuff that would be a mess to handle with the traditional way of writing a custom rc script per service.
The problem is not being able to write in languages like ruby or python. It's to have to write something at all.
systemd makes many things declarative that were historically procedural, potentially painful to debug, code.
This eases distro maintenance and I suppose is one of the top reasons most distros adopted it.
wrt the Unix philosophy, discussions about it related to systemd are often (always?) too abstract to be useful, I'd suggest talking about specific problematic points.
It's funny how many people fall in love with the Unix philosophy because they enjoy using an OS with a macrokernel that ships with awk, tar, and find, which they operate with useless uses of cat.
Macrokernel or microkernel or a grain of salt, whatever kernel you use does not matter when it comes to seeing the advantages of 'the *nix philosophy'. As to the 'useless uses of cat' these often make the pipe easier to grasp because the first step is always the same:
A kernel is a very special program, and splitting it into individual components would be orthogonal to the Unix philosophy, which is about user space programs. Besides, Linux is quite modular, and only loads what it needs, so the fact that it's monolithic is not a major concern. Yes, it would be better if kernel panics wouldn't impact the system, but nowadays these are very rare, and are usually related to hardware rather than software issues.
As for GNU utils and the examples you mention, those indeed align with the Unix philosophy, which you clearly misunderstand.
SUSE is using btrfs as the default filesystem in all their offerings. I think that SLES is less widespread than RHEL, but I also think those people at SUSE know what they are doing. It's not like we hear of massive data loss from SLES environments all the time.
Even the much-maligned raid5/6 is basically fine†.
If you're concerned about the write hole, use -m DUP/raid1/raid1c2 instead of -m raid5. Plus raid-stripe-tree†† is coming - didn't check the status of it recently.
Many horror stories are because, while btrfs is fine, the operational model and tooling have some footguns which can cause either straight up data loss (due to operator error, but arguably that's really due to bad UX) or possible-but-hard-to-get-out-of situations.
I use btrfs because using zfs has been painful for me, for two reasons:
- btrfs can "shapeshift": I progressively moved _live_ from 2hdd raid1 to 5hdd raid5 data + raid1c2 meta with varying experiments in between. Probably five or six rebalance to change its shape over the years.
- the zfs module situation: when I tried it, the module silently failed to build properly and this resulted in a broken system til I fixed it; this happened twice over six months. Luckily I anticipated this failure mode and only the data array (not the root fs) was zfs, so I could still boot and login into a full system to fix.
Compared to zfs, btrfs is slow to scrub and rebalance though.
I used btrfs for most of the 2010's. I nearly lost some data once, and was able to recover it. Afterwards, I moved to ZFS and never looked back.
btrfs may be great now, and more power to people who use it and are happy. However, I am so used to the ergonomics of ZFS (and zed, and ZFS integrated encryption) that I don't see a reason to migrate back.
I've lost my laptops SSD (as in: no longer mountable, only got data out of it with some rescue tools) at some point between 2017 and 2020, don't remember when exactly. I've also had a weird experience where a btrfs filesystem formatted on my desktop PC was not mountable on a Raspberry Pi, and vice versa formatted on the Pi was not mountable on the desktop. That didn't instill confidence either.
On the other hand, I've been running a btrfs RAID1 on two HGST datacenter drives for a few years and haven't had issues with that.
for features declared stable it's been an urban legend for a long time.
I use BTRFS in prod since I think 2016 which was also the last year I lost data to an BTRFS Bug
For me Linux distributions have always been a way to have cheap UNIX like systems at home, any POSIX like system will do, it could equally been one of the BSDs or Windows NT/POSIX, had Microsoft been actually serious about it.
Actually I was more found of Solaris, Irix, NeXTSTEP in how they approached the whole development experience.
Still got some nice memories of Aix and HP-UX as well, with Xenix and DG/UX as introductory experiences to the UNIX world.
Something like Android is closer to how Plan 9/Inferno got to be, than most GNU/Linux distros, regarding a managed userspace, and more interesting to see where it all goes.
Or modern approaches like Unikernels (even if POSIX based), managed runtimes on top of type 1 hypervisors, immutable container based OSes,....
Also commercial UNIXes never followed the "Unix philosophy" that keeps being endless recited in Linux circles, ironically, given that GNU tools are hardly anything to go by, given the endless list of options they have available.
Windows is going downhill in a hurry, while MacOS is stagnated (at best), while Linux is constantly advancing, slowly, maybe sometime the wrong path, but always moving.
That's a curious take. Today's Linux distributions are more reliable than ever with more long-term support than ever.
What changed is that you usually do not run a snowflake anymore which you carefully update to the next version in situ, but some amount of compute and storage. Today everything is blue-green and updates mean deploy, destroy behind a load balancer.
I was a UNIX developer (kernel, as well as userland programs like troff) for 7 years, working for the first company to provide commercial support. (Our founder, the former head of the Cornell CompSci department, while at RAND, had them make legal arrangements for Western Electric to provide RAND with a commercial license for $20,000 for the no-support tape that they provided to universities for $300 ... from there the commercial UNIX world took off.) Later I started my Linux life with the Yggdrasil distro, my first of many. Nowadays, in retirement, I do development on WSL on Windows. (Getting a working and convenient emacs environment on it was a bit of effort but now it's how I do all of my editing.)
> It is your device that is not compatible with Linux
No truer words, and it is very hard to get people to understand that phrase. Like the author I tried to get people on-board with Linux in the 90's, but it was a very hard time. No one switched, and considering I worked on a large programming group, to my surprise the people who heard of it was very small.
After IBM did its thing in 2000, a couple of years later these same people would ask me questions about it, but no one switched. My manager even had me do a demo on it at work.
But the direction of Linux is worrying me. But I have a second older Laptop with a BSD on it, and that will give me an out if necessary. Sadly that may happen sooner then later :(
Readit News
I understand the downsides people have of systemd, but I have the feeling the huge upside is often overlooked.
> but I have the feeling the huge upside is often overlooked.
It is fine to objectively compare trade-offs. However had, it has to be a fair comparison; we can not start with "init manager" because systemd does a lot more, so how can a comparison to any software with less code be fair then? runit doesn't do much more than for initializing.
The objection to it is not to it as an init manager. To quote the description from the systemd site:
> systemd is a suite of basic building blocks for a Linux system
I remember embracing systemd funny enough due to when I was running alpine as my server and I had to write my own r script and boy did that quickly make me remember the awful times on freebsd debugging sh.
I don't want my init system to replace cron. I don't want it to manage logging. I don't want it to have any debugging capabilities. All of these things can be done with other programs, arguably in a much more flexible and robust way.
An init system should do one thing (well): manage system services. Within that context, it should start services on boot, keep them running in the background, and allow the user to create, stop, and start services. That's it. And it could be argued that even those responsibilities are too broad for a single program.
So I understand that you and many others like that systemd provides solutions to all of these tasks in an integrated way. But you should also understand that this does in fact go against the Unix philosophy, of small, independent, but composable programs with a single responsibility. It "proposes" alternatives to many other tools for no particular reason, until users are effectively using GNU/systemd/Linux.
And, yes, I know that technically systemd is not a monolith and is composed of many programs, but that is a moot point. It is a single project, maintained by a strongly opinionated team, and given the importance of it, most distros go all-in, so users are strongly recommended, if not forced, to use all of its programs. In many cases it's not even possible to use one individual systemd program idependently from another. This is why systemd is seen as a kraken that takes over the entire system if you do decide to use one part of it.
One does not need cron or systemd for scheduling tasks.
- You claim that the Unix philosophy only survives to the GNUtils. Well, that shows to me a lack of understanding what the philosophy is about. Everything is a file is similar to the OOP approach of everything is an object. I recommend watching Ken Thompson when he was young here: https://www.youtube.com/watch?v=tc4ROCJYbm0
It does not capture all the UNIX philosophy but it does extend on the reasons why that philosophy works well. The philosophy is bigger than that of course but it helps serve as a counter-argument.
- The example of "writing your own script" is no different to a non-systemd system. Why would a script work or not work based on systemd? You mention as example FreeBSD debugging a shell script. Well, others use proper languages such as ruby or python. Everything that can be done via systemd I can do without it too and, in fact, have been doing so. Ruby essentially runs my system as the primary layer on everything (granted, it runs Linux, and thus mostly C, and ruby is at the end of the day a syntactic wrapper over C). I never understood why systemd would matter. I read the advertisement of the systemd devs - none of this applied to my use cases, so I never "embraced" systemd, simply because I never needed it. I did point out the increased complexity of it as a negative trade-off and this has been true til this day.
- Former "hater" also implies that criticism is not based on rationale and logic. This is not the case either. It's funny to me how the pro-systemd camp isn't really able to come up with compelling arguments on their own.
I would say your comment either shows a lack of understanding, or that you completely missed the point.
> The example of "writing your own script" is no different to a non-systemd system. Why would a script work or not work based on systemd?
Of course, you can write the service itself in Python or Ruby or whatever regardless the service manager. The point is that with systemd, or upstart, or other service managers like this that make things more declarative (launchd?), you don't have to write a script to manage the service at all.
On systemd, you declare which services yours depend on, how to run it, which user should be used to run your service, and many things are handled for you, including many security mechanisms you don't need to think about and provide further config for this stuff that would be a mess to handle with the traditional way of writing a custom rc script per service.
The problem is not being able to write in languages like ruby or python. It's to have to write something at all.
systemd makes many things declarative that were historically procedural, potentially painful to debug, code.
This eases distro maintenance and I suppose is one of the top reasons most distros adopted it.
wrt the Unix philosophy, discussions about it related to systemd are often (always?) too abstract to be useful, I'd suggest talking about specific problematic points.
No.
As for GNU utils and the examples you mention, those indeed align with the Unix philosophy, which you clearly misunderstand.
> Even if your btrfs, after almost 18 years, still eats data in spectacular fashion.
Is this (by now) an urban legend ? Is btrfs any less reliable than, say, xfs/ext4 etc. nowadays ?
If you're concerned about the write hole, use -m DUP/raid1/raid1c2 instead of -m raid5. Plus raid-stripe-tree†† is coming - didn't check the status of it recently.
Many horror stories are because, while btrfs is fine, the operational model and tooling have some footguns which can cause either straight up data loss (due to operator error, but arguably that's really due to bad UX) or possible-but-hard-to-get-out-of situations.
I use btrfs because using zfs has been painful for me, for two reasons:
- btrfs can "shapeshift": I progressively moved _live_ from 2hdd raid1 to 5hdd raid5 data + raid1c2 meta with varying experiments in between. Probably five or six rebalance to change its shape over the years.
- the zfs module situation: when I tried it, the module silently failed to build properly and this resulted in a broken system til I fixed it; this happened twice over six months. Luckily I anticipated this failure mode and only the data array (not the root fs) was zfs, so I could still boot and login into a full system to fix.
Compared to zfs, btrfs is slow to scrub and rebalance though.
† https://unixdigest.com/articles/battle-testing-zfs-btrfs-and...
†† https://lore.kernel.org/linux-btrfs/cover.1698679287.git.dst...
btrfs may be great now, and more power to people who use it and are happy. However, I am so used to the ergonomics of ZFS (and zed, and ZFS integrated encryption) that I don't see a reason to migrate back.
On the other hand, I've been running a btrfs RAID1 on two HGST datacenter drives for a few years and haven't had issues with that.
Actually I was more found of Solaris, Irix, NeXTSTEP in how they approached the whole development experience.
Still got some nice memories of Aix and HP-UX as well, with Xenix and DG/UX as introductory experiences to the UNIX world.
Something like Android is closer to how Plan 9/Inferno got to be, than most GNU/Linux distros, regarding a managed userspace, and more interesting to see where it all goes.
Or modern approaches like Unikernels (even if POSIX based), managed runtimes on top of type 1 hypervisors, immutable container based OSes,....
Also commercial UNIXes never followed the "Unix philosophy" that keeps being endless recited in Linux circles, ironically, given that GNU tools are hardly anything to go by, given the endless list of options they have available.
What changed is that you usually do not run a snowflake anymore which you carefully update to the next version in situ, but some amount of compute and storage. Today everything is blue-green and updates mean deploy, destroy behind a load balancer.
Deleted Comment
No truer words, and it is very hard to get people to understand that phrase. Like the author I tried to get people on-board with Linux in the 90's, but it was a very hard time. No one switched, and considering I worked on a large programming group, to my surprise the people who heard of it was very small.
After IBM did its thing in 2000, a couple of years later these same people would ask me questions about it, but no one switched. My manager even had me do a demo on it at work.
But the direction of Linux is worrying me. But I have a second older Laptop with a BSD on it, and that will give me an out if necessary. Sadly that may happen sooner then later :(