I was a systemd hater. I agree with a lot of the post but in practice I need a whole lot less custom scripts, forking code, etc to maintain a simple service whether it’s a custom c binary, go, rust, nodejs , ruby , python… I don’t need to mess with monit(for most things) and yeah in less words then my post here I could have a working service that would both restart on crash and on reboot…. Do I miss the init.d scripts and writing pid files forking twice to change user … no
No idea why software tries to "daemonize" anyway. Seems to be some kind of weird tradition. It's always been the wrong thing to do. They should just run normally and output to standard streams even if it's connected to a terminal. The reasons are outlined here:
Ironically, systemd is one of few service managers which can properly track those weird "daemonizing" processes. It uses Linux's cgroups. It doesn't matter how many times they fork, they can't hide from systemd.
Not sure what exactly you were pointing to, but section "Doing it right" (4.3.1) recommends:
mydaemon & daemonpid=$!
This is a terrible advice - that's how you end up with all sorts of bad things:
- you ssh'd into server, restarted server, checked it worked. Then you logged out and server got killed.
- your server runs as root or power user
- your server output/error goes somewhere, not clear where
- your server's process got paused by SIGTTOU, because it was trying to write to console
etc...
The complex daemonization dance, with double-forks, closing random FDs, etc.. was required to avoid all this bad stuff before systemd days. Sysvinit/openrc folks came up with "start-stop-daemon" helper, but it was pretty terrible in general, especially in logging department (it had none).
Unfortunately, there aren't any concrete examples (unless I skipped something accidentally), so it's hard to tell what exactly is meant as "does more" or that it doesn't "stand out of the way" etc.
Yes, systemd services can do a lot of things. Except that before rc scripts did all those things, but without any unified and standardized primitives. systemd provides those for a lot of common use cases - and that's how it ended up doing so much - because our rc scripts used to do that. systemd actually does less because unit files are not scripts (but one can hook any external program in a lot of places, if they need to).
Also, as I understand it, systemd isn't a single piece of software either, it's a whole project with a bunch of semi-autonomous pieces, tied with common conventions and assumptions. It's difficult to define the scope for an umbrella project, and I believe individual pieces all have their scopes relatively well-defined.
>> The single, overarching problem with systemd is that it attempts, in every possible way, to do more instead of less.
The more systemd does, the better I like the power, features, consistency and functionality my linux systems have.
systemd is a beautiful work of art.
Software developers owe it to themselves to full understand what systemd has to offer, because if you really understand it then you'll find you can architect your systems around systemd and often let it to the hard work for you.
If anything, the old gnu/linux name should be dropped and replaced with linux/systemd to illustrate that in fact systemd is really a major part of what Linux is.
This reminds me of teenagers of the last generations who preferred distorted mp3 sound to original because that's the sound they have grew up with. Teenagers of today prefer distorted wide-angle selfies instead of properly shot faces because again that's what they have grew up with.
Calling systemd a work of art gives me the same vibe.
I can't tell if it's sarcasm or genuine, but I 90% agree. There's a lot of things that it makes easier. But there are areas which are just weird and overengineered and overcomplicated - for example resolved interacting with old files and creating lots of possible ways it can run. If you look at forums / SO it's obvious people are confused about what it tries to achieve. Instead of creating a whole new system, they could've started by implementing and standardizing the multi-interface behaviour in established projects like dnsmasq. Instead of that they went with NIH where it really wasn't necessary.
For resolve.conf, I think it is fundamentally bad design that is hard to get right. Before systemd, I've spent plenty of time trying to get networkmanager and resolvconf to play together. And then there was those dhcp scripts which would edit the files directly.
Also, what do you mean my "standardizing dnsmasq behaviour?" From my recollection of using it (and from manpage[0]), it's pretty basic. You (1) manually edit /etc/resolve.conf to point to localhost dnsmasq instance (2) use app-specific ways to configure nameservers, in dnsmasq's case it's an ad-hoc list of secondary resolv.conf files specified on command line, with no defaults. I don't think this is worth standardizing, as this scheme cannot even represent network interface -> dns entry associations.
> If you look at forums / SO it's obvious people are confused about what it tries to achieve.
Well, many people are confused about how DNS resolving works. Many of do not consider a problem, when they configure their resolv.conf to two different upstreams, each returning different answers (i.e. one responding for domains that the other doesn't) and then wondering, why stuff is broken.
Of course that's a bit of a troll comment at the end there. There are plenty of Linux distros still not using systemd. https://www.devuan.org/ as one example.
Trying to compare Windows updates favorably to updates on systemd seems a bit bizarre. I can't ever remember seeing an upgrade ask for a reboot on Raspberry PI OS (debian 12). And I can't ever remember seeing a Windows update that didn't reboot the system.
As a long-time Windows developer (both apps and device drivers) substantially coming to Linux long after the decision to use systemd was made, I am impressed as heck by Systemd.
And was completely horrified by my very limited experience, as a user, with pre-systemd Linuxes. That was insane. Manually edit what in to which system file? And my system invariably didn't even HAVE that file.
Even writing systemd units is virtually effortless.
I get frustrated when people misstate the Unix philosophy.
The Unix philosophy is about function, not form. Say you have a tool to send something over a network. In 1993, that tool can get by being quite simple. In 2024, that same tool now needs to be more complex, just to do the same 'one job' and to 'do it well'.
It's a consistent insistence that the 'single job' be defined today just as it was in 1980 that results in Linux requiring hundreds of packages to be even remotely usable. Oh, you want Wi-Fi? Well, that's not really the job of the networking stack, the job of the networking stack is to connect PDP-11s together...
> Oh, you want Wi-Fi? Well, that's not really the job of the networking stack, the job of the networking stack is to connect PDP-11s together...
WiFi absolutely should be a separate component; the unix philosophy is an excellent defense against servers in a datacenter trying to install a WiFi stack.
wifi happens to be a separate component; wpa_supplicant or iwl are separate user space tools that network management tools call into; on the kernel side, the drivers and their stack are separate modules.
The Unix philosophy is about function, not form. Say you have a tool to send
something over a network. In 1993, that tool can get by being quite simple.
In 2024, that same tool now needs to be more complex, just to do the same
'one job' and to 'do it well'.
Even by that standard scope remains a problem with systemd, it makes emacs look svelte. It's absorbed big pieces like time management and logging that ought not be part of a purported init system.
A core part of the Unix philosophy is being able to pipe standard streams between processes, files, sockets, etc, as the user sees fit.
Systemd, particularly journald, violates this severely by snarfing any stream you want to send to /var/log or wherever and locking it in a binary database somewhere, that you have to use journalctl to access.
I have no problem with journald as an idea, but the audacity that it knows better than the user what to do with stdout/stderr is insane, and 100% anti-Unix.
> In 1993, that tool can get by being quite simple. In 2024, that same tool now needs to be more complex, just to do the same 'one job' and to 'do it well'.
1993's sendmail is more complex than qmail. The latter uses a modular system, where each subtool does its job well, rather than sendmail's monolith design.
Linux was(is?) the system. Shell scripts could configure the system, reasonably well. Systemd meant it was no longer possible to simply fix a linux distro that had it...a massive recompile and intricate understanding of many low level systems was needed.
No, you shouldn't need to know about PDP-11s to fix your buggy wifi, neither should you need to know a massive binary subsystem to configure your wifi either...
So as far as doing its "one job", it did/does it fairly poorly, even today. It is a lot more standard and it does get points from me in that respect, but it fails at being a configurable system completely.
I don't think systemd is meant to be fixed or modified, unless you are actually contributing to the upstream systemd project and have accepted that it will be a time consuming project.
It seems to be made for, and works well for, a modern ephemeral infrastructure workflow.
Build stuff to work with an unmodified distro, wipe it and reinstall if it goes wrong.
I was indifferent to systemd until podman shipped quadlet. Now being able to bring up containers, tied to the system, in a reliable and efficient manner, without docker compose or big bespoke commands, has been incredible
Every time systemd comes up everyone says it's great and how much better it is than SysVinit. No one EVER talks about the actual alternatives that people use like OpenRC, runit, s6.
By all means, be the change you want to see. All I know about OpenRC is that it doesn't support user services the same way systemd does. I know even less about runit or s6, so I'd love to hear more from people who use them in their daily computing. Because unfortunately the impression I get from never seeing anyone talk about OpenRC, Shepherd, Upstart, et cetera; is that either nobody uses them, or they don't add anything worth noting over SysVinit.
Though I'm surprised that you say:
> Every time systemd comes up everyone says it's great and how much better it is
When my experience is usually that people complain about systemd and it being over-engineered and doing too much and having too many components (yes, I know that those projects are related to systemd, not part of it, I'm just saying what I see people mention as downsides).
OpenRC just works for me (Alpine) there’s a declarative side that’s more than enough for simple services, but you can override logic for starting, stopping, reloading,… as it’s just a shell script.
> unfortunately the impression I get from never seeing anyone talk about OpenRC, Shepherd, Upstart, et cetera; is that either nobody uses them
People do use them. Multiple distros either default to one of those, or support it: Alpine, Devuan, MX Linux, Slackware, Void, Guix, Gentoo and others.
> or they don't add anything worth noting over SysVinit.
Not looked at this myself, but do they mostly not add something - dependencies, easier config or something?
Readit News
No idea why software tries to "daemonize" anyway. Seems to be some kind of weird tradition. It's always been the wrong thing to do. They should just run normally and output to standard streams even if it's connected to a terminal. The reasons are outlined here:
https://mywiki.wooledge.org/ProcessManagement
Ironically, systemd is one of few service managers which can properly track those weird "daemonizing" processes. It uses Linux's cgroups. It doesn't matter how many times they fork, they can't hide from systemd.
- you ssh'd into server, restarted server, checked it worked. Then you logged out and server got killed.
- your server runs as root or power user
- your server output/error goes somewhere, not clear where
- your server's process got paused by SIGTTOU, because it was trying to write to console
etc...
The complex daemonization dance, with double-forks, closing random FDs, etc.. was required to avoid all this bad stuff before systemd days. Sysvinit/openrc folks came up with "start-stop-daemon" helper, but it was pretty terrible in general, especially in logging department (it had none).
-I can spawn/respawn daemons as someone other than root. SysV inittab runlevels are root-only.
-I used runlevel 4 for all my magic. Now I can have so much more magic.
-nspawn has let me craft containers from day 1.
-Path units are so much better than inotifywait (and somewhat better than incron).
-Do I prefer socket units to inetd? Mostly.
-Free automounter! Yes, crazy syntax, but gratis!
Do I wish it worked on BSD? Yes.
It IS functional - it does things that are helpful and important.
But the implementation is hard to swallow. It is just so far away from elegant, understandable and usable.
Yes, systemd services can do a lot of things. Except that before rc scripts did all those things, but without any unified and standardized primitives. systemd provides those for a lot of common use cases - and that's how it ended up doing so much - because our rc scripts used to do that. systemd actually does less because unit files are not scripts (but one can hook any external program in a lot of places, if they need to).
Also, as I understand it, systemd isn't a single piece of software either, it's a whole project with a bunch of semi-autonomous pieces, tied with common conventions and assumptions. It's difficult to define the scope for an umbrella project, and I believe individual pieces all have their scopes relatively well-defined.
The more systemd does, the better I like the power, features, consistency and functionality my linux systems have.
systemd is a beautiful work of art.
Software developers owe it to themselves to full understand what systemd has to offer, because if you really understand it then you'll find you can architect your systems around systemd and often let it to the hard work for you.
If anything, the old gnu/linux name should be dropped and replaced with linux/systemd to illustrate that in fact systemd is really a major part of what Linux is.
This reminds me of teenagers of the last generations who preferred distorted mp3 sound to original because that's the sound they have grew up with. Teenagers of today prefer distorted wide-angle selfies instead of properly shot faces because again that's what they have grew up with.
Calling systemd a work of art gives me the same vibe.
Also, what do you mean my "standardizing dnsmasq behaviour?" From my recollection of using it (and from manpage[0]), it's pretty basic. You (1) manually edit /etc/resolve.conf to point to localhost dnsmasq instance (2) use app-specific ways to configure nameservers, in dnsmasq's case it's an ad-hoc list of secondary resolv.conf files specified on command line, with no defaults. I don't think this is worth standardizing, as this scheme cannot even represent network interface -> dns entry associations.
https://thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html
Well, many people are confused about how DNS resolving works. Many of do not consider a problem, when they configure their resolv.conf to two different upstreams, each returning different answers (i.e. one responding for domains that the other doesn't) and then wondering, why stuff is broken.
As a long-time Windows developer (both apps and device drivers) substantially coming to Linux long after the decision to use systemd was made, I am impressed as heck by Systemd.
And was completely horrified by my very limited experience, as a user, with pre-systemd Linuxes. That was insane. Manually edit what in to which system file? And my system invariably didn't even HAVE that file.
Even writing systemd units is virtually effortless.
The Unix philosophy is about function, not form. Say you have a tool to send something over a network. In 1993, that tool can get by being quite simple. In 2024, that same tool now needs to be more complex, just to do the same 'one job' and to 'do it well'.
It's a consistent insistence that the 'single job' be defined today just as it was in 1980 that results in Linux requiring hundreds of packages to be even remotely usable. Oh, you want Wi-Fi? Well, that's not really the job of the networking stack, the job of the networking stack is to connect PDP-11s together...
I counted no less then 64 related executables related to systemd on my system.
Accusing systemd of "not doing one thing and doing it well" is like accusing the GNU project of not following the Unix philosophy.
WiFi absolutely should be a separate component; the unix philosophy is an excellent defense against servers in a datacenter trying to install a WiFi stack.
Deleted Comment
Systemd, particularly journald, violates this severely by snarfing any stream you want to send to /var/log or wherever and locking it in a binary database somewhere, that you have to use journalctl to access.
I have no problem with journald as an idea, but the audacity that it knows better than the user what to do with stdout/stderr is insane, and 100% anti-Unix.
1993's sendmail is more complex than qmail. The latter uses a modular system, where each subtool does its job well, rather than sendmail's monolith design.
Linux was(is?) the system. Shell scripts could configure the system, reasonably well. Systemd meant it was no longer possible to simply fix a linux distro that had it...a massive recompile and intricate understanding of many low level systems was needed.
No, you shouldn't need to know about PDP-11s to fix your buggy wifi, neither should you need to know a massive binary subsystem to configure your wifi either...
So as far as doing its "one job", it did/does it fairly poorly, even today. It is a lot more standard and it does get points from me in that respect, but it fails at being a configurable system completely.
What kinds of problems are you hitting where you need to recompile systemd?
It seems to be made for, and works well for, a modern ephemeral infrastructure workflow.
Build stuff to work with an unmodified distro, wipe it and reinstall if it goes wrong.
Though I'm surprised that you say:
> Every time systemd comes up everyone says it's great and how much better it is
When my experience is usually that people complain about systemd and it being over-engineered and doing too much and having too many components (yes, I know that those projects are related to systemd, not part of it, I'm just saying what I see people mention as downsides).
People do use them. Multiple distros either default to one of those, or support it: Alpine, Devuan, MX Linux, Slackware, Void, Guix, Gentoo and others.
> or they don't add anything worth noting over SysVinit.
Not looked at this myself, but do they mostly not add something - dependencies, easier config or something?