You can use systemd-run with --shell (or a subset of options enabled by --shell) and -p to specify service properties to run commands interactively in a similar environment as your service.
This can help troubleshoot issues and makes experimenting with systemd options faster.
I think there's been some talk about adding a built-in way for systemd-run to copy settings out of a .service file, but it doesn't exist yet.
I've written Perl/Python scripts to do this for me. They're not really aimed at working with arbitrary services, but it should be possible to adapt to different scenarios.
There are some gotchas I ran into. For example, with RuntimeDirectory: systemd deletes the directory once the process exits, even if there's still another process running with the same RuntimeDirectory value set.
It's also really useful for doing parallel builds of modules that may actually consume all available memory when you can't force the build system to use fewer cores than you have available.
Both in terms of artificially reducing the number of CPUs you expose, but also in terms of enforcing a memory limit that will kill all processes in the build before the broader kernel OOM killer will act, in case you screw up the number of CPUs.
woah that's actually awesome.
I feel like adding uh storage usage limits could also be easy as well.
But the one thing that I always wonder is about (virtualization?) in the sense of something like docker just for containerizing or some sort of way of running them in some sort of sandbox without much performance issues or something, I am kinda interested in knowing what might be the best way of doing so (is podman the right way or some other way like bubblewrap?)
Edit: just discovered in the comment below the (parents parents?)comment that there is systemd isolation too, that sounds very interesting and the first time I personally heard of it hmm
Another issue I just ran into is that a colon separated value for ExecSearchPath doesn't work in systemd-run/-p. You have to specify each path as a separate -p argument.
There are some minor annoyances like that, but it's not too hard to work around.
I feel like Docker and other containerization tools are becoming even less relevant given that systemd can twiddle the same isolation bits so there's no real difference in terms of security that using a container tool grants.
Seeing that podman can run containers as systemd services (see https://codesmash.dev/why-i-ditched-docker-for-podman-and-yo... ), it seems like using containers other than as a distribution mechanism has few advantages, and many disadvantages in terms of dependency updates requiring container rebuilds.
> I feel like Docker and other containerization tools are becoming even less relevant given that systemd can twiddle the same isolation bits so there's no real difference in terms of security that using a container tool grants.
I see it as _exactly_ the opposite. Podman gives me more or less the same security controls as systemd and the package/delivery problem is solved.
Call me when `systemctl pull ...` fetches the binary and everything else needed to run it _and_ puts the .service file in the right spot.
nixos kind of does that except better. Usually just set services.foo.enabled to true along with any other config you want. It's also super easy to wrap services in a container if you want, and doing so is kept conceptually separate from dependency management. If you want to make your own systemd service, then referencing a package in `ExecStart` or whatever will make it automatically get pulled in as a dependency.
Container rebuilds are disadvantages?
Using mkosi and systemd-nspawn for containers it doesnt really feel that way, still a lot easier to build some distroless app container than to finangle a service to have zero access to other binaries, libraries, or other data entirely.
I dont get the distribution "advantage" building em with mkosi but I'd argue it a weakness as far too many are running containers with who-knows-what inside them.
Docker is absolutely less relevant. My personal machines haven't run Docker for months and my employer is finishing our migration away from Docker in a few months.
Docker/podman can be an NPM left-pad solution. Ideally you skip a lot of work by using prebuilt docker files. I think there are times when it is just as easy and safer to use a systemd unit file. Ofcourse it depends a lot on the application. Its a whole lot easier to distribute your own static binary than a closed source app linked to a bunch of ancient libraries.
What makes me scratch my hand is why the failed access violations are not easy to show and log. A correctly configured service should not attempt to access things is is not intended to access. If it has to check if it has access and act conditionally, this also should be made explicit, either in the service code, or in its configuration.
There should be an strace-like tool that would collect a log of such "access denied" erros for troubleshooting. Even better, each service should run in its own process group, and tracing could be switched on / off for a particular process group.
> A correctly configured service should not attempt to access things is is not intended to access. If it has to check if it has access and act conditionally
It's normally recommended to attempt the access and handle the denial, instead of doing two separate steps (checking for access and doing the access); the later can lead to security issues (https://en.wikipedia.org/wiki/TOCTOU).
The fact that systemd continues to get hate, ~15 years after mass adoption, is a cultural phenomenon worth understanding. Benno Rice of freebsd gave a super interesting talk about this: The Tragedy of systemd: https://www.youtube.com/watch?v=o_AIw9bGogo
I can only imagine how long the Wayland haters will be writing blogs once LTS distro start shipping Wayland-first desktops. Looking at the whole upstart/systemd drama, I'm guessing we'll hit the 2k38 bug before they'll find something new to write about.
By whose definition of sanity? systemd haters often believe that it is sysvinit that was sane while it is systemd who is insane. I am one of those people but being systemd hater I'm obviously wrong because old.
Dude’s been arguing with people since at least 2012 that systemd is a good thing. It took me less than a minute to figure that out by searching his blog.
PulseAudio also drew a lot of disapproval, until Pipewire appeared and finally did the same thing (and more) well.
Maybe systemd (service management, logind, the DNS resolver, the loggig system, etc) will eventually be re-implemented in a way that does not have the irritating properties of the original systemd.
/* I'd say that systemd feels like typical corporate software. It has a ton of features to check all the requisite boxes, it's not very ergonomic, it does things the way authors wanted and could sell to the corporate customers (who are not the end users), not the way end users prefer. It also used to be bug-ridden for quite some time, after having been pushed upon users. It comes from Red Hat (which is now a part of IBM), so you could say: here's why! But, say, podman also comes from Red Hat, and does not feel like such corporate software; to the contrary, end users enjoy it. */
Hey, now I am interested in more of such softwares overall.
Like imagine a list where we can create a form where people can give them and give reasonings or just something.
What if I can create a github repo and issues regarding this so that we can discuss about them and I can create a website later if it interests but its a really nice thought experiment.
Are we talking more about uh every software including proprietory too?
Are we talking about lets say websites too or services (what if we extend it to services like websites or even products outside of software niche into things beyond too, that's interesting too)
Another interesting point that comes to my mind might be that cryptocoins might be the lowest inverse to this software project in the sense that I believe that there was very little net positive done to all humanity in general, sure the privacy aspects are nice but still, its not worth having people invest their life savings into it thinking that its going to 100x y'know, I have created a whole article about it being frustated by this idea people think regarding crypto as an investment when it could very well be a crypto"currency" but that's a yap for another day.
I really nerded over this and I think I loved it, we need a really good discussion about it :>
I think I agree. I’m curious what software would be in places 2-10. If we’re talking about HN, maybe excel/google sheets? Maybe C++? Recent versions of macOS always seem to get hate, but I think macOS is in a different category.
> The bad news [...] with the assumption that DNS resolution is being done via systemd-resolved
This is incredible, really.
systemd-resolved is easily the worst piece that came out of systemd crowd. I see systemd-resolved as a transparent film that protects the shiny parts of new devices: something that is meant to be removed immediately after unboxing. This, and timesyncd.
Now, it's no wonder systemd crowd now want this thing mandatory. They truly want to be noticed and the best way to be noticed is to make someone suffer while virtue signaling righteousness.
The private /tmp strike us, when update to Debían 12 servers and find that a batch process cannot access the same temporal files that our web application.
Luckily, it's very easy to fix, adding an extra systems file to disable that feature on the Tomcat service.
I don't understand how people consider this article "systemd hate".
The article is informative. Even a bit bland when it comes to opinion on the matter of systemd as a whole. The article is literally just saying "if you write services, complain loudly and with context about permission errors" and "if you use systemD with hardening enabled, consider it alongside discretionary access control and mandatory access control when troubleshooting permissions errors".
Readit News
This can help troubleshoot issues and makes experimenting with systemd options faster.
I think there's been some talk about adding a built-in way for systemd-run to copy settings out of a .service file, but it doesn't exist yet.
I've written Perl/Python scripts to do this for me. They're not really aimed at working with arbitrary services, but it should be possible to adapt to different scenarios.
https://gist.github.com/dextercd/59a7e5e25b125d3506c78caa3dd...
There are some gotchas I ran into. For example, with RuntimeDirectory: systemd deletes the directory once the process exits, even if there's still another process running with the same RuntimeDirectory value set.
Both in terms of artificially reducing the number of CPUs you expose, but also in terms of enforcing a memory limit that will kill all processes in the build before the broader kernel OOM killer will act, in case you screw up the number of CPUs.
But the one thing that I always wonder is about (virtualization?) in the sense of something like docker just for containerizing or some sort of way of running them in some sort of sandbox without much performance issues or something, I am kinda interested in knowing what might be the best way of doing so (is podman the right way or some other way like bubblewrap?)
Edit: just discovered in the comment below the (parents parents?)comment that there is systemd isolation too, that sounds very interesting and the first time I personally heard of it hmm
Another issue I just ran into is that a colon separated value for ExecSearchPath doesn't work in systemd-run/-p. You have to specify each path as a separate -p argument.
There are some minor annoyances like that, but it's not too hard to work around.
Seeing that podman can run containers as systemd services (see https://codesmash.dev/why-i-ditched-docker-for-podman-and-yo... ), it seems like using containers other than as a distribution mechanism has few advantages, and many disadvantages in terms of dependency updates requiring container rebuilds.
I see it as _exactly_ the opposite. Podman gives me more or less the same security controls as systemd and the package/delivery problem is solved.
Call me when `systemctl pull ...` fetches the binary and everything else needed to run it _and_ puts the .service file in the right spot.
importctl pull-tar https://example.com/image.tar.gz && portablectl attach image
https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
That would be pretty awesome, actually.
I dont get the distribution "advantage" building em with mkosi but I'd argue it a weakness as far too many are running containers with who-knows-what inside them.
Do you work in the software industry?
Containers are as relevant as ever, of course.
Deleted Comment
There should be an strace-like tool that would collect a log of such "access denied" erros for troubleshooting. Even better, each service should run in its own process group, and tracing could be switched on / off for a particular process group.
It's normally recommended to attempt the access and handle the denial, instead of doing two separate steps (checking for access and doing the access); the later can lead to security issues (https://en.wikipedia.org/wiki/TOCTOU).
Some references:
- https://docs.arbitrary.ch/security/systemd.html
- https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db...
"People who hate person X are often a masterclass in finding fault in wonderful, intelligent, faithful, generous men."
People who talk like this are worse than systemd.
Maybe systemd (service management, logind, the DNS resolver, the loggig system, etc) will eventually be re-implemented in a way that does not have the irritating properties of the original systemd.
/* I'd say that systemd feels like typical corporate software. It has a ton of features to check all the requisite boxes, it's not very ergonomic, it does things the way authors wanted and could sell to the corporate customers (who are not the end users), not the way end users prefer. It also used to be bug-ridden for quite some time, after having been pushed upon users. It comes from Red Hat (which is now a part of IBM), so you could say: here's why! But, say, podman also comes from Red Hat, and does not feel like such corporate software; to the contrary, end users enjoy it. */
Like imagine a list where we can create a form where people can give them and give reasonings or just something.
What if I can create a github repo and issues regarding this so that we can discuss about them and I can create a website later if it interests but its a really nice thought experiment.
Are we talking more about uh every software including proprietory too?
Are we talking about lets say websites too or services (what if we extend it to services like websites or even products outside of software niche into things beyond too, that's interesting too)
Another interesting point that comes to my mind might be that cryptocoins might be the lowest inverse to this software project in the sense that I believe that there was very little net positive done to all humanity in general, sure the privacy aspects are nice but still, its not worth having people invest their life savings into it thinking that its going to 100x y'know, I have created a whole article about it being frustated by this idea people think regarding crypto as an investment when it could very well be a crypto"currency" but that's a yap for another day.
I really nerded over this and I think I loved it, we need a really good discussion about it :>
This is incredible, really.
systemd-resolved is easily the worst piece that came out of systemd crowd. I see systemd-resolved as a transparent film that protects the shiny parts of new devices: something that is meant to be removed immediately after unboxing. This, and timesyncd.
Now, it's no wonder systemd crowd now want this thing mandatory. They truly want to be noticed and the best way to be noticed is to make someone suffer while virtue signaling righteousness.
But please do tell us why it's garbage
[1] https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-ac...
The article is informative. Even a bit bland when it comes to opinion on the matter of systemd as a whole. The article is literally just saying "if you write services, complain loudly and with context about permission errors" and "if you use systemD with hardening enabled, consider it alongside discretionary access control and mandatory access control when troubleshooting permissions errors".