Podman was good when it supported systemd unit files, so I could auto start and auto update containers, even entire pods with systemd.
Then they removed that in favor of Quadlet. Now in order to do a single container I can do a unit file, but for a pod, I need to use a Kubernetes cluster definition.
Plus, unlike Docker their containers bow to SELinux definitions, so I have repeatedly struggled with containers unable to access mapped directories.
So what is it, Podman? Should I just use Kubernetes? Should I just make dedicated directories for everything instead of mapping logical places for things?
I already was defining my infrastructure with docker-compose.yml files, and found out that podman-compose has a poorly documented feature that generates systemd units. It doesn't use the now-deprecated podman feature, it writes the unit files itself, and I find the process much smoother than the podman feature anyway.
Its documented. If you just type `podman-compose` in the command line you get...
usage: podman-compose [-h] [-v] [--in-pod in_pod] [--pod-args pod_args]
[--env-file env_file] [-f file] [-p PROJECT_NAME]
[--podman-path PODMAN_PATH] [--podman-args args]
[--podman-pull-args args] [--podman-push-args args]
[--podman-build-args args] [--podman-inspect-args args]
[--podman-run-args args] [--podman-start-args args]
[--podman-stop-args args] [--podman-rm-args args]
[--podman-volume-args args] [--no-ansi] [--no-cleanup]
[--dry-run]
{help,version,wait,systemd,pull,push,build,up,down,ps,run,exec,start,stop,restart,logs,config,port,pause,unpause,kill}
...
command:
{help,version,wait,systemd,pull,push,build,up,down,ps,run,exec,start,stop,restart,logs,config,port,pause,unpause,kill}
...
systemd create systemd unit file and register its compose stacks
When first installed type `sudo podman-compose systemd -a create-unit`
later you can add a compose stack by running `podman-compose systemd -a register`
then you can start/stop your stack with `systemctl --user start podman-compose@<PROJ>`
> unlike Docker their containers bow to SELinux definitions,
That's a bug in docker. If your system isn't configured for SELinux, disable it.
Also the systemd files generated by podman-generate-systemd are just executing "podman start containername", you can write them on your own easily but (unlike e.g. docker-composr) the containers are black boxes pretty much.
The advantage of quadlet is that the definition of the container is declared in the .container file; before I used to write the podman run command line manually in a handwritten systemd unit, and quadlet is a big improvement in that respect and can be an alternative to docker-compose (with advantages and disadvantages).
I keep forgetting this transition happened until I try to 'podman generate systemd [...]'
This is rare because I wrote an Ansible role to do this in a way that feels nice.
Anyway, it really feels like podman lost the mark. I've already subscribed to the unit file maintenance/relationship planning thing. Just let me use the generator. I don't care about Quadlets or how they might be better.
I recently migrated over to NixOS which treats systemd as the source of truth for everything, including containers. I found this model extremely intuitive, but it was difficult to apply this to Docker Compose without a lot of manual migration. So I ended up writing a tool that handles this for you — it converts your Compose files into a NixOS config that can be interpreted and managed natively.
Arion can wrap docker-compose and run as a project or part of a nixos config. Did you come across Arion before creating this, and have you compared them?
I had a brief look through your examples and it doesn’t look like compose2nix implements docker-compose’s network per compose file. Is this something you want to add?
That was a cool feature (I didn't realize it was gone, that's unfortunate), although I felt the generated code wasn't super great, and if the container is stateless (excepting what's stored in volumes of course) then it's so simple to write your own systemd unit file that I just do that now. I wrote it once and pretty much just copy/paste it when needed, changing the podman run command for image names, port numbers, volumes, etc. For example, here's what I use for Jellyfin. Just drop at `/etc/systemd/system/jellyfin.service`:
Note: You can also just `s/podman/docker/g` and reuse the same service file with docker, which is really convenient for systems where you have no choice
> So what is it, Podman? Should I just use Kubernetes?
If you're talking about a production system for any business larger than a 10 person tech startup; yeah, probably. Alternatively there's Docker Swarm and Hashicorp Nomad. Though Swarm is not nearly as flexible, it's just easy to use. And Nomad... well, let's just say I've been paying closer attention to Hashicorp's build processes in their open source repos like Packer and Vault as of late and they do some stuff that seems shady to me so use at your own risk.
`podman generate systemd` is still there, and I see no reason you couldn't use it. it's just a bunch of podman commands wrapped in a unit file, no magic.
feels like a lot more cruft than quadlets to me though.
Mapping directories from the host requires that you change selinux labels on those files so that the container process can access the files.
That's just how selinux works.
I almost never see what is IMHO the killer feature of Podman touted as a reason to prefer it over Docker: Docker mangles your network config. It is a nightmare trying to run Docker and KVM virtual machines with bridges at the same time. Podman on the other hand plays very nice OOTB.
I've also had a lot of VPNs break and/or be broken by Docker. I don't know much about the way podman does networking, but whatever it is they did a good job thinking it through and it has yet to interfere with anything else I do. I definitely can't say the same for Docker
Buildah (which may well work with Docker, but is a Podman peer) is the killer feature in my opinion. Dockerfiles are unadulterated shite. One of my pet peeves is "bored developers" writing DSLs/programming languages (especially in YAML, but that isn't the case here) when an off-the-shelf language would have done. Dockerfiles are a genuinely fantastic example of why this nonsense needs to end.
To see why, look at Buildah without `bud`. Instead of a silly DSL that becomes annoying the second your use-case veers an inch off the happy path, you can use Bash, or Fish, or whatever it is that you want.
These types of bad decisions carry the into the rest of the Docker ecosystem. DCS and it's (perpetually incomplete) replacement are yet more examples: instead of using established signing protocols (like Cosign does) they had the desire to build an obnoxiously complex and hard to automate (especially key rotation) system.
Glad that podman is getting more traction tbh, too many tools are built with the assumption that people add the sudo docker group which break if you have any kind of security conscious docker setup (such as not blindly giving it root access).
I'm trying to get up to speed with docker atm, is it the applications inside the container that are typically running as root (which people are careless about from a "whats the worst that can happen" attitude?) or is the typical pattern that people are running the docker host process as root (maybe in order to allow the host to expose ports at 80/443 without some reverse proxy/firewall in front of it?)
As a certified RHEL engineer I have been using Podman for years already.
To be perfectly honest I do enjoy it for all my personal container use. But at work I still use docker for our developers. There is so far nothing I can offer our developers that can match docker compose in simplicity.
We even use buildah in CI pipelines when we make container images, but specifically for developer end users docker compose is still dominant.
Should, but there are still edge cases where people use weird edge features of Docker's implementation that don't always work out of the box with Podman (it's a lot better now, though, and most of my compose files are fine either way).
Docker even just messes with iptables in its default configuration/installation. It's always been a pain point especially if one wants to use the newer nftables.
I've got a cryptominer in one of my personal selfhosted container of a github project meant to be used with a vpn only, due to this insecure by default choice of docker. Still salty about it...
Speaking of networking issues, one big problem I ran into was running out of memory for network devices when using a nested setup (requiring a system restart to fix). Would have been a great lxc alternative otherwise.
I still don't really understand why Red Hat invests into creating a Docker alternative, but I really like it. Podman does pretty much everything Docker does, but it has more features (e.g. pods) or the way Podman does it tends to be better (e.g. daemonless container spawning process).
The main issue to a common developer would be Docker compose I suppose, which if you use simple compose files, there's actually a podman-compose script that attempts to be compatible with the Docker compose spec.
There's also using Podman as a backend for docker-compose [1]. Overall, in 2024, I see no reason using Docker at least on Linux boxes. Not sure how Podman fares on macOS or Windows.
> I still don't really understand why Red Hat invests into creating a Docker alternative, but I really like it.
They originally tried to work together with docker to fix various issues (e.g. systemd compatibility for certain use case) which had popped up in the past but this turned out not very fruit full.
If you combine this that docker had and still has a uncanny amount of security issues(1) and docker like containers/images being very widely used by developers they had very little choice then to create their own implementation which is more in line with the values and approaches of RHEL.
(1): Like docker still not defaulting to rootless even through it can. For a hardened context both the the docker user group and the ways to run it without the group are security wise a no-go in many use-cases. Like docker initially doing way to little to make sure non privileged containers are at least somewhat sandboxed and taking forever to fix it even when it became known it's an issue. Like the way it interacts with firewalls and networks rules. Like the way it interacts with SELinux. Etc. In companies with dedicated Linux system administrators which care about security docker being banned is not that rare.
I use Podman on Windows. Docker on Windows has been an endless stream of annoyances and frustration for me, starting from the fact everything points you to install an aggressively user-unfriendly GUI that starts up heavyweight processes on boot, nags you to sign up for a cloud account for some reason, and then nags you again that you shouldn't be using this so-called open source product for work anyway. Installing "just the command line" docker as an unsupported alternative is unnecessarily complicated, and last time I tried most WSL VMs didn't even support it out of the box anyway.
Enter Podman, you just winget install podman and it gets out of your way. When you need to run a container, you start up the Podman VM. When you don't, your system behaves exactly as it always did before. If you need to run something in a compose file, podman-compose is there. Maybe it can't handle esoteric configs, but it's worked for my use cases. The only thing Podman doesn't do is integrate with VS Code properly, but honestly the struggle of getting Docker to behave itself on Windows is far more annoying to me than missing some keyboard shortcuts in VS Code, so for me it's not a problem.
I would (and do) recommend Podman as the default solution for containers on Windows. The only reason I can see to use Docker is if your company is paying for it, or if you have a complicated compose setup, in which case it might be worth moving to Kubernetes anyway.
They invest in it because Docker is waaay out of spec with how things are done on Linux. Doing things that break the system, struggling with rootless for years, and who can miss the vendor lock-in? Podman is open and compliant and compliments k8s as well. It’s just nonsense the amount of effort developers have invested in Docker because it was first to market for easy containers.
Docker is every bit as open and not locked in as podman. Perhaps even more so as it's so widely used and doesn't require redhat specific projects around it. Are you confusing docker with docker hub?
docker isn't really struggeling with rootles, it works with rootless since a long time
they just never bothered making it the default or officially supporting it
which in context with how they acted about other security problems int the past tells a lot about how serious the docker company takes security on linux
> I see no reason using Docker at least on Linux boxes. Not sure how Podman fares on macOS or Windows.
I hope the majority doesn't end up with your point of view. Docker is not RedHat/IBM. If Docker goes away RedHat gets to continue to push their corporate agenda with a heavier hand. There are some advantages to Podman, but there are also some things that have not been executed well in RedHat's mission to replace all things Docker. If you look at the history of how RedHat approached the container space they could have contributed, and improved, a lot of projects they ultimately wasted time and effort on. RedHat isn't a company that has an agenda of doing right by the customer and that's why you should consider what you hope for from a bigger picture perspective.
Red Hat had ample reason to develop Podman when they started the project and no reason now to abandon it.
Red Hat tried to work with Docker, but it didn’t go well. Docker also shot itself in the foot trying to push Swarm over Kubernetes and a bunch of other silliness until they had some serious management/leadership changes.
Podman, IIRC, is 100% FOSS. Not sure you can say the same for Docker. If Red Hat gets stupid with Podman, the rest the community can pick it up and carry on under a new name. Not true of Docker. If Docker gets bought by, say Broadcom tomorrow, the community can only fork the bits that are open.
I use Podman on Mac OS. I've found the experience better than Docker for the most part, especially when it comes to supporting older versions of Mac OS. The only major downside of Podman on Mac OS is that you cant use the host network in a container, but people will rarely want to use the host network in a container anyway. This is something to keep in mind if you want to experiment with network things in a container and want to retain the same hostname and IP address of your host, however. I manage to work around this limitation anyway.
> The only major downside of Podman on Mac OS is that you cant use the host network in a container
And this isn't even a Podman-specific issue, it's true of Docker Desktop as well [0]:
> The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.
I'm a podman beginner, trying to install ollama-webui(1) using Podman on M2 MBA.
I started up Podman Desktop, and did a terminal command "docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v ollama-webui:/app/backend/data --name ollama-webui --restart always ghcr.io/ollama-webui/ollama-webui:main" based on Github's instructions, but it gave a error message something about "host".
the exact error message was ""Error": "failed to create new hosts file: unable to replace \"host-gateway\" of host entry \"host.docker.internal:host-gateway\": host containers internal IP address is empty""
Do you know what is the problem and how do I overcome this?
If I run the above command using Docker Desktop, it runs and installs Ollama-WebUI just fine.
Sounds good, I like their security first approach and some of the decisions they've made, going for secure defaults out of the box, and that it works with docker compose. I wonder if podman gains enough traction, at some point, they decide to go their own way with regards to the commands and the yml, because right now it seems to be a tool that 'hangs on to' docker and docker's compose file format.
It would be good to have a swarm alternative in podman, it seems like k8s is a crutch for lack of orchestration. With their good security hat on they could probably come up with a sane, simple way of running containers at small scales without having to dive into a PhD in k8s which doesn't have secure defaults out of the box, while maintaining compatibility with docker compose format.
Anyway that's a good intro thanks for sharing, I'll be trying it later.
I agree that rootless containers and isolated namespaces are critical security features. But with docker rootless, this is also possible and not complicated. You just have to do it. I have written a blog post to set up Mastodon in docker rootless with all the best practices currently available [1].
The benefit with sticking with docker is that accessibility is better: More communities, more blogs, broad availability of docker compose configs, more peers knowing how to use it etc. In the end, both podman and docker run processes in isolated namespaces on the host.
Readit News
Then they removed that in favor of Quadlet. Now in order to do a single container I can do a unit file, but for a pod, I need to use a Kubernetes cluster definition.
Plus, unlike Docker their containers bow to SELinux definitions, so I have repeatedly struggled with containers unable to access mapped directories.
So what is it, Podman? Should I just use Kubernetes? Should I just make dedicated directories for everything instead of mapping logical places for things?
To enable the feature:
To register a systemd unit: Updating looks like this: ($PROJECT_NAME is usually the directory name.)Source code for the feature is here if you care to inspect it:
https://github.com/containers/podman-compose/blob/f6dbce3618...
Admittedly, I'm still on podman 4.3.1, but I don't see any reason why this would stop working in later versions of podman.
That's a bug in docker. If your system isn't configured for SELinux, disable it.
Also the systemd files generated by podman-generate-systemd are just executing "podman start containername", you can write them on your own easily but (unlike e.g. docker-composr) the containers are black boxes pretty much.
The advantage of quadlet is that the definition of the container is declared in the .container file; before I used to write the podman run command line manually in a handwritten systemd unit, and quadlet is a big improvement in that respect and can be an alternative to docker-compose (with advantages and disadvantages).
I feel obligated to say that you should set it to permissive mode, never disable SElinux.
I keep forgetting this transition happened until I try to 'podman generate systemd [...]'
This is rare because I wrote an Ansible role to do this in a way that feels nice.
Anyway, it really feels like podman lost the mark. I've already subscribed to the unit file maintenance/relationship planning thing. Just let me use the generator. I don't care about Quadlets or how they might be better.
Add the following to containers.conf if you don't want to deal with it:
If you don't like podman's default security level, there is usually a way to turn things off.https://github.com/aksiksi/compose2nix
Arion can wrap docker-compose and run as a project or part of a nixos config. Did you come across Arion before creating this, and have you compared them?
I had a brief look through your examples and it doesn’t look like compose2nix implements docker-compose’s network per compose file. Is this something you want to add?
Isn't that great? You want SELinux if you deploy containers on prod.
> I have repeatedly struggled with containers unable to access mapped directories.
Use the `z` or `Z` flag, e.g. `podman run -v mydir:podman_dir:z nginx` to get SELinux labels updated automatically (and temporarily) by podman.
If you're talking about a production system for any business larger than a 10 person tech startup; yeah, probably. Alternatively there's Docker Swarm and Hashicorp Nomad. Though Swarm is not nearly as flexible, it's just easy to use. And Nomad... well, let's just say I've been paying closer attention to Hashicorp's build processes in their open source repos like Packer and Vault as of late and they do some stuff that seems shady to me so use at your own risk.
feels like a lot more cruft than quadlets to me though.
Mapping directories from the host requires that you change selinux labels on those files so that the container process can access the files. That's just how selinux works.
Deleted Comment
It's just a simplified layer ontop of those old systemd units.
I've also had a lot of VPNs break and/or be broken by Docker. I don't know much about the way podman does networking, but whatever it is they did a good job thinking it through and it has yet to interfere with anything else I do. I definitely can't say the same for Docker
To see why, look at Buildah without `bud`. Instead of a silly DSL that becomes annoying the second your use-case veers an inch off the happy path, you can use Bash, or Fish, or whatever it is that you want.
These types of bad decisions carry the into the rest of the Docker ecosystem. DCS and it's (perpetually incomplete) replacement are yet more examples: instead of using established signing protocols (like Cosign does) they had the desire to build an obnoxiously complex and hard to automate (especially key rotation) system.
I'm doing it right now, no nightmares, just works, odd.
Very cool.
To be perfectly honest I do enjoy it for all my personal container use. But at work I still use docker for our developers. There is so far nothing I can offer our developers that can match docker compose in simplicity.
We even use buildah in CI pipelines when we make container images, but specifically for developer end users docker compose is still dominant.
I almost got burned by this.
The main issue to a common developer would be Docker compose I suppose, which if you use simple compose files, there's actually a podman-compose script that attempts to be compatible with the Docker compose spec.
There's also using Podman as a backend for docker-compose [1]. Overall, in 2024, I see no reason using Docker at least on Linux boxes. Not sure how Podman fares on macOS or Windows.
[1] https://www.redhat.com/sysadmin/podman-docker-compose
They originally tried to work together with docker to fix various issues (e.g. systemd compatibility for certain use case) which had popped up in the past but this turned out not very fruit full.
If you combine this that docker had and still has a uncanny amount of security issues(1) and docker like containers/images being very widely used by developers they had very little choice then to create their own implementation which is more in line with the values and approaches of RHEL.
(1): Like docker still not defaulting to rootless even through it can. For a hardened context both the the docker user group and the ways to run it without the group are security wise a no-go in many use-cases. Like docker initially doing way to little to make sure non privileged containers are at least somewhat sandboxed and taking forever to fix it even when it became known it's an issue. Like the way it interacts with firewalls and networks rules. Like the way it interacts with SELinux. Etc. In companies with dedicated Linux system administrators which care about security docker being banned is not that rare.
Enter Podman, you just winget install podman and it gets out of your way. When you need to run a container, you start up the Podman VM. When you don't, your system behaves exactly as it always did before. If you need to run something in a compose file, podman-compose is there. Maybe it can't handle esoteric configs, but it's worked for my use cases. The only thing Podman doesn't do is integrate with VS Code properly, but honestly the struggle of getting Docker to behave itself on Windows is far more annoying to me than missing some keyboard shortcuts in VS Code, so for me it's not a problem.
I would (and do) recommend Podman as the default solution for containers on Windows. The only reason I can see to use Docker is if your company is paying for it, or if you have a complicated compose setup, in which case it might be worth moving to Kubernetes anyway.
they just never bothered making it the default or officially supporting it
which in context with how they acted about other security problems int the past tells a lot about how serious the docker company takes security on linux
I hope the majority doesn't end up with your point of view. Docker is not RedHat/IBM. If Docker goes away RedHat gets to continue to push their corporate agenda with a heavier hand. There are some advantages to Podman, but there are also some things that have not been executed well in RedHat's mission to replace all things Docker. If you look at the history of how RedHat approached the container space they could have contributed, and improved, a lot of projects they ultimately wasted time and effort on. RedHat isn't a company that has an agenda of doing right by the customer and that's why you should consider what you hope for from a bigger picture perspective.
Red Hat tried to work with Docker, but it didn’t go well. Docker also shot itself in the foot trying to push Swarm over Kubernetes and a bunch of other silliness until they had some serious management/leadership changes.
Podman, IIRC, is 100% FOSS. Not sure you can say the same for Docker. If Red Hat gets stupid with Podman, the rest the community can pick it up and carry on under a new name. Not true of Docker. If Docker gets bought by, say Broadcom tomorrow, the community can only fork the bits that are open.
they tried to work with docker, it didn't work out, that is why we have podman
and given the headache "securely" using docker is I'm really happy that they did
After the first two or so sentences, you don't mention a single specific thing in your comment, so it's difficult to understand what you mean.
And this isn't even a Podman-specific issue, it's true of Docker Desktop as well [0]:
> The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.
[0] https://docs.docker.com/network/drivers/host/
I started up Podman Desktop, and did a terminal command "docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v ollama-webui:/app/backend/data --name ollama-webui --restart always ghcr.io/ollama-webui/ollama-webui:main" based on Github's instructions, but it gave a error message something about "host".
the exact error message was ""Error": "failed to create new hosts file: unable to replace \"host-gateway\" of host entry \"host.docker.internal:host-gateway\": host containers internal IP address is empty""
Do you know what is the problem and how do I overcome this?
If I run the above command using Docker Desktop, it runs and installs Ollama-WebUI just fine.
Thank you.
(1) https://github.com/ollama-webui/ollama-webui ("Installing with Docker")
It would be good to have a swarm alternative in podman, it seems like k8s is a crutch for lack of orchestration. With their good security hat on they could probably come up with a sane, simple way of running containers at small scales without having to dive into a PhD in k8s which doesn't have secure defaults out of the box, while maintaining compatibility with docker compose format.
Anyway that's a good intro thanks for sharing, I'll be trying it later.
The benefit with sticking with docker is that accessibility is better: More communities, more blogs, broad availability of docker compose configs, more peers knowing how to use it etc. In the end, both podman and docker run processes in isolated namespaces on the host.
[1]: https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless...