Much better article with very real tips about what options to try than yesterday's (weirdly flagged/dead?) post on the topic. Which while I really enjoyed lacked substance; I was in the comments trying to provide a more useful basis with some real examples, but this is an exemplary list of awesome ways systemd can easily quickly readily provide aassive boost to isolation & security. Great write up!
Nitpick and title correction: The proper spelling of systemd is systemd, not SystemD. According to their brand page:
Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. Why? Because it's a system daemon, and under Unix/Linux those are in lower case, and get suffixed with a lower case d. And since systemd manages the system, it's called systemd.
Many of the more juvenile systemd haters used to deliberately spell it that way.
As an insult, it was rather less successful than the "Micro$oft" / "Slowaris" / "HP-SUX" mockery from the 1990s - but it did manage to sow enough confusion that that it still pops up regularly today, even in contexts that are otherwise neutral or positive about it.
Thanks for sharing this. It looks like you can also use "systemd-analyze" with the "--user" flag to inspect systemd user units as well ("systemd-analyze --user security"). I've started using systemd more now that I've transitioned my containers to Podman, and this will be a helpful utility to improve the security of my systemd unit/container services.
You can ofcourse achieve all these things in your init scripts which are unique in their way and not uniform at all, just to give credit where credit is due. But systemd makes it practical to use our beloved kernel and it's features in an uniform and standard way... :)
I started my Linux journey so late I can't imagine living without systemd, the few systems I've encountered without systemd are such a major PITA to use.
I recently discovered "unshare" which I could use to remount entire /nix RW for some hardlinking shenanigans without affecting other processes.
systemd is so good, warty UX when interacting with it but the alternative is honestly Windows in my case.
Aside, I'm quite surprise that systemd won out, given all the hate for so many years. You got nearly every forum and HN thread that even tangently mention systemd, flooded with people proclaiming systemd to be the spawn of the devil "forced down their throat". At the same time I noticed that a lot of regular Linux users (like developers that just want to deploy stuff) seemed to like systemd unit files. I guess all those complainers, however vocal, were neither the majority, nor the people who actually maintained distributions.
This feeling is particular striking for me, because I once worked on a Linux project with the aim of improving packaging and software distribution. We also got a lot of hate, mainly for not being .deb or.rpm, and it looked to me as if the hate was a large reason for the failure of the project.
systemd is very complex software. Alternative is very simple software with complex scripting which will reimplement parts of systemd in a buggy way (and that's not necessarily a bad thing). systemd probably is inspired by Windows and other service managers, while old sysv init is just a tiny launcher for script tree.
Just an example of systemd limitation is that systemd does not support musl, so if you want to build a tiny embedded sysroot, you already have some limitations.
Why don't distros flip more of these switches? Are there cons of being more aggressive with these settings? It's really a lot for many people to tinker with.
> Are there cons of being more aggressive with these settings?
well, the con is you might unknowingly break some setups. take NetworkManager: after tightening it down, did you check both IPv4 and IPv6 connectivity? did you check that both the `dns=systemd-resolved` and `dns=default` modes of operation (i.e. who manages /etc/resolv.conf) work? did you check its ModemManager integrations, that it can still manage cellular connections? did you check that the openvpn and cisco anyconnect plugins still work? what about the NetworkManager-dispatcher hooks?
> Why don't distros flip more of these switches?
besides the bit of "how many distro maintainers actually understand the thing they're maintaining well enough to know which switches can be flipped without breaking more than 0.01% of user setups", there's the bit of "should these flags be owned by the distro, or by the upstream package?" if the distro manages these, they'll get more regressions on the next upstream release. if the upstream sets these, they can't be as aggressive without breaking one or two of their downstreams.
This question is sorta similar to "Why don't distros enable restrictive MAC policies by default"
Maintainers _could_ take the time to lock down sshd and limit the damage it can do if exploited, but there are costs associated with that:
1. Upfront development cost
2. Maintenance cost from handling bug reports (lots of edge cases for users)
3. Maintenance cost from keeping this aligned with upstream changes
You could extend this argument and say that distros shouldn't bother with _any_ security features, but part of the job of a distro maintainer is to strike a balance here, and similar to SELinux / AppArmor / whatever, most mainstream desktop distro maintainers probably don't think the juice is worth the squeeze.
> Which distro has the best out-of-the-box output for?:
systemd-analyze security
desbma/shh generates SyscallFilter and other systems unit rules from straces similar to how audit2allow generates SELinux policies by grepping for AVC denials in permissive mode (given kernel parameters `enforcing=0 selinux=1`), but should strace be installed in production?:
I don't know but I would imagine the settings are sometimes new. Not everyone is a systemd nerd, and even if they were you may actually break stuff for old versions of systemd if you try to turn them on.
We could also ask why nobody seems to use SELinux or AppArmor, or any other random security feature. Most distros have these things available but most developers and users are not familiar, don't truly need it, etc.
It would be cool to have a repo with suggested hardening for common services, since there's so many different hardening options. One of the things you might notice from lots of users using common suggestions, is that the permissions often need to be opened up more than you'd think, to support edge cases.
> It would be cool to have a repo with suggested hardening for common services
From packaging stuff for nixpkgs, a distro that often is without upstream support, it is usually very useful to look at how mainstream distro package services.
Those hardening steps also tend to be well tested even if sometimes a bit lax. If you want to find out how, e. G., postgresql can be hardened, consider looking at the Debian, Ubuntu and/or RHEL packages as a starting point.
Distros don't usually do security hardening, unless the distro is security-specific. They slap something generic on like AppArmor or SELinux and call it a day. (This article is the proof of that... all the default services are not hardened). Usually this is a good thing, as it prioritizes usability, and lets the user harden as they wish.
these Hardening variables have been discussed some years back[1].
this will not take off I'm afraid, because locking these unitfiles down is offloaded to the end-user (I've yet to see maintainers embrace shipping locked down files). Maybe they will? But this same approach hasn't worked with apparmor so why should it work with systemd? Who will do the job?
If you consider apparmor maintainers provide skeleton-templates in many cases that will make the parser stop complaining. ("look I have a profile so apparmor shuts up, but don't take too close a look OK")
Then there is firejail, which some argue[2] is snake-oil considering the high level of administrative glue compared to its massive attack-surface (also it's a setuid binary).
I didn't mention SElinux since I don't know a single person who had the joy (or pain depending on perspective) of working with it. But again, seems the expectation to implement security with it is shifted to the user.
> this will not take off I'm afraid, because locking these unitfiles down is offloaded to the end-user (I've yet to see maintainers embrace shipping locked down files).
I vaguely recall looking at the slides from a talk on OpenBSD's approach to this topic, which came down to (paraphrasing from hazy memory) "if it can be disabled, people will disable it; if it needs to be configured, people won't configure it".
> this will not take off I'm afraid, because locking these unitfiles down is offloaded to the end-user
Maybe your point is that this isn't done by the vendor in practice. And I'm sure there's room for lots of improvement. However, one of the great things about how systemd units can be provided by the vendor and seamlessly tweaked by the administrator is that the vendor (i.e. packager and/or distro) can set these up easily.
There definitely are packages that ship with locked-down files. Tor and powerdns (pdns) are two off the top of my head.
→ Overall exposure level for pdns.service: 1.9 OK
→ Overall exposure level for tor.service: 7.1 MEDIUM
I think it should be done by the maintainer of the software not by the distro. My concern is that these features are available since at least 5 years and it has not yet caught on (regardless of what this blog article recommends).
It would be great to see it implemented but for now at least on Debian/sid the situation is as follows:
UNIT EXPOSURE PREDICATE
ModemManager.service 6.3 MEDIUM
NetworkManager.service 7.8 EXPOSED
alsa-state.service 9.6 UNSAFE
anacron.service 9.6 UNSAFE
atop.service 9.6 UNSAFE
atopacct.service 9.6 UNSAFE
avahi-daemon.service 9.6 UNSAFE
blueman-mechanism.service 9.6 UNSAFE
bluetooth.service 6.0 MEDIUM
cron.service 9.6 UNSAFE
dbus.service 9.3 UNSAFE
dictd.service 9.6 UNSAFE
dm-event.service 9.5 UNSAFE
dnscrypt-proxy.service 8.1 EXPOSED
emergency.service 9.5 UNSAFE
exim4.service 6.9 MEDIUM
getty@tty1.service 9.6 UNSAFE
irqbalance.service 1.2 OK
lvm2-lvmpolld.service 9.5 UNSAFE
polkit.service 1.2 OK
rc-local.service 9.6 UNSAFE
rescue.service 9.5 UNSAFE
rtkit-daemon.service 7.2 MEDIUM
smartmontools.service 9.6 UNSAFE
systemd-ask-password-console.service 9.4 UNSAFE
systemd-ask-password-wall.service 9.4 UNSAFE
systemd-bsod.service 9.5 UNSAFE
systemd-hostnamed.service 1.7 OK
systemd-journald.service 4.9 OK
systemd-logind.service 2.8 OK
systemd-networkd.service 2.9 OK
systemd-timesyncd.service 2.1 OK
systemd-udevd.service 7.1 MEDIUM
tor@default.service 6.6 MEDIUM
udisks2.service 9.6 UNSAFE
upower.service 2.4 OK
user@1000.service 9.4 UNSAFE
wpa_supplicant.service 9.6 UNSAFE
It would be nice to be possible to do the hardening of services via allowlisting instead. E.g. AllowNothing=true and then start adding what is allowed to make the service function.
Readit News
https://github.com/desbma/shh
EDIT: More info here: https://github.com/systemd/systemd/issues/33688
Yesterday's, just in case: https://us.jlcarveth.dev/post/hardening-systemd.mdhttps://news.ycombinator.com/item?id=44928504
Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. Why? Because it's a system daemon, and under Unix/Linux those are in lower case, and get suffixed with a lower case d. And since systemd manages the system, it's called systemd.
As an insult, it was rather less successful than the "Micro$oft" / "Slowaris" / "HP-SUX" mockery from the 1990s - but it did manage to sow enough confusion that that it still pops up regularly today, even in contexts that are otherwise neutral or positive about it.
I started my Linux journey so late I can't imagine living without systemd, the few systems I've encountered without systemd are such a major PITA to use.
I recently discovered "unshare" which I could use to remount entire /nix RW for some hardlinking shenanigans without affecting other processes.
systemd is so good, warty UX when interacting with it but the alternative is honestly Windows in my case.
This feeling is particular striking for me, because I once worked on a Linux project with the aim of improving packaging and software distribution. We also got a lot of hate, mainly for not being .deb or.rpm, and it looked to me as if the hate was a large reason for the failure of the project.
Just an example of systemd limitation is that systemd does not support musl, so if you want to build a tiny embedded sysroot, you already have some limitations.
well, the con is you might unknowingly break some setups. take NetworkManager: after tightening it down, did you check both IPv4 and IPv6 connectivity? did you check that both the `dns=systemd-resolved` and `dns=default` modes of operation (i.e. who manages /etc/resolv.conf) work? did you check its ModemManager integrations, that it can still manage cellular connections? did you check that the openvpn and cisco anyconnect plugins still work? what about the NetworkManager-dispatcher hooks?
> Why don't distros flip more of these switches?
besides the bit of "how many distro maintainers actually understand the thing they're maintaining well enough to know which switches can be flipped without breaking more than 0.01% of user setups", there's the bit of "should these flags be owned by the distro, or by the upstream package?" if the distro manages these, they'll get more regressions on the next upstream release. if the upstream sets these, they can't be as aggressive without breaking one or two of their downstreams.
Maintainers _could_ take the time to lock down sshd and limit the damage it can do if exploited, but there are costs associated with that:
You could extend this argument and say that distros shouldn't bother with _any_ security features, but part of the job of a distro maintainer is to strike a balance here, and similar to SELinux / AppArmor / whatever, most mainstream desktop distro maintainers probably don't think the juice is worth the squeeze.From https://news.ycombinator.com/item?id=29995566 :
> Which distro has the best out-of-the-box output for?:
desbma/shh generates SyscallFilter and other systems unit rules from straces similar to how audit2allow generates SELinux policies by grepping for AVC denials in permissive mode (given kernel parameters `enforcing=0 selinux=1`), but should strace be installed in production?:desbma/shh: https://github.com/desbma/shh
We could also ask why nobody seems to use SELinux or AppArmor, or any other random security feature. Most distros have these things available but most developers and users are not familiar, don't truly need it, etc.
From packaging stuff for nixpkgs, a distro that often is without upstream support, it is usually very useful to look at how mainstream distro package services.
Those hardening steps also tend to be well tested even if sometimes a bit lax. If you want to find out how, e. G., postgresql can be hardened, consider looking at the Debian, Ubuntu and/or RHEL packages as a starting point.
this will not take off I'm afraid, because locking these unitfiles down is offloaded to the end-user (I've yet to see maintainers embrace shipping locked down files). Maybe they will? But this same approach hasn't worked with apparmor so why should it work with systemd? Who will do the job?
If you consider apparmor maintainers provide skeleton-templates in many cases that will make the parser stop complaining. ("look I have a profile so apparmor shuts up, but don't take too close a look OK")
Then there is firejail, which some argue[2] is snake-oil considering the high level of administrative glue compared to its massive attack-surface (also it's a setuid binary).
I didn't mention SElinux since I don't know a single person who had the joy (or pain depending on perspective) of working with it. But again, seems the expectation to implement security with it is shifted to the user.
[1] https://news.ycombinator.com/item?id=22993304
[2] https://github.com/netblue30/firejail/issues/3046
https://fedoraproject.org/wiki/Changes/SystemdSecurityHarden...
Maybe your point is that this isn't done by the vendor in practice. And I'm sure there's room for lots of improvement. However, one of the great things about how systemd units can be provided by the vendor and seamlessly tweaked by the administrator is that the vendor (i.e. packager and/or distro) can set these up easily.
There definitely are packages that ship with locked-down files. Tor and powerdns (pdns) are two off the top of my head.
It would be great to see it implemented but for now at least on Debian/sid the situation is as follows:
and really, these should be written by the developers, not distro maintainers
poking around on my Ubuntu machine, a few daemons have some hardening, chronyd looks pretty good
[0] https://github.com/jart/pledge