Back in 2001/2002, I was charged with building a WiFi hotspot box. I was a fan of OpenBSD and wanted to slim down our deployment, which was running on Python, to avoid having to copy a ton of unnecessary files to the destination systems. I also wanted to avoid dependency-hell. Naturally, I turned to `chroot` and the jails concept.
My deployment code worked by running the software outside of the jail environment and monitoring the running processes using `ptrace` to see what files it was trying to open. The `ptrace` output generated a list of dependencies, which could then be copied to create a deployment package.
This worked brilliantly and kept our deployments small and immutable and somewhat immune to attack -- not that being attacked was a huge concern in 2001 as it is today. When Docker came along, I couldn't help but recall that early work and wonder whether anyone has done a similar thing to monitor file usage within Docker containers and trim them down to size after observing actual use.
The best CI/CD pipeline I ever used was my first freelance deployment using Django. I didn't have a clue what I was doing and had to phone a friend.
We set up a git post receive hook which built static files and restarted httpd on a git receive. Deployment was just 'git push live master'.
While I've used Docker a lot since then, that remains the single easiest deployment I've ever had.
I genuinely don't understand what docker brings to the table. I mean, I get the value prop. But it's really not that hard to set up http on vanilla Ubuntu (or God forbid, OpenBSD) and not really have issues.
Is the reproducibility of docker really worth the added overhead of managing containers, docker compose, and running daemons on your devbox 24/7?
> I genuinely don't understand what docker brings to the table. I mean, I get the value prop. But it's really not that hard to set up http on vanilla Ubuntu (or God forbid, OpenBSD) and not really have issues.
Sounds great if you're only running a single web server or whatever. My team builds a fairly complex system that's comprised of ~45 unique services. Those services are managed by different teams with slightly different language/library/etc needs and preferences. Before we containerized everything it was a nightmare keeping everything in sync and making sure different teams didn't step on each others dependencies. Some languages have good tooling to help here (e.g. Python virtual environments) but it's not so great if two services require a different version of Boost.
With Docker, each team is just responsible for making sure their own containers build and run. Use whatever you need to get your job done. Our containers get built in CI, so there is basically a zero percent chance I'll come in in the morning and not be able to run the latest head of develop because someone else's dev machine is slightly different from mine. And if it runs on my machine, I have very good confidence it will run on production.
> I genuinely don't understand what docker brings to the table. I mean, I get the value prop. But it's really not that hard to set up http on vanilla Ubuntu (or God forbid, OpenBSD) and not really have issues.
For me, as an ex-ops, the value proposition is to be able to package a complex stack made of one or more db, several services and tools (ours and external), + describe the interface of these services with the system in a standard way (env vars + mounts points).
It massively simplify the onboarding experience, make updating the stack trivial, and also allow devs, ci and prod to run the same version of all the libraries and services.
Not having to regularly rebuild the whole dev environment because I need to work on one particular Python app once a quarter and its build chain reliably breaks other stuff? Priceless.
Someone wrote a PHP7 script to generate some of our daily reports a while back that nobody wants to touch. Docker happily runs the PHP7 code in the container and generates the reports on any system. its portable, and it doesnt require upkeep.
Docker in and of itself does not do you much good. Its strength comes from the massive amounts of generic tooling that is built around the container as the standard deployable unit.
If you want to handle all your deployments the same way, you can basically only choose between Nix and containers. Unfortunately, containers are far more popular and have more tooling.
> Is the reproducibility of docker really worth the added overhead of managing containers, docker compose, and running daemons on your devbox 24/7?
Yes. Everything on my box is ephemeral and can be deleted and recreated or put on another box with little-to-no thought. Infrastructure-as-code means my setup is immutable and self-documented.
It's a little more time to set up initially, but now I know exactly what is running.
I don't really understand the 24/7 comment, now that it is set up there's very very little maintenance. Sometimes an upgrade might go askew but that is rare.
Any change to it is recorded as a git commit, I don't have to worry about logging what I've done ever because it's done for me.
Changes are handled by a GitHub action, all I have to do to change what is running is commit a file, and the infra will update itself.
I don't use docker-compose, I use a low-overhead microk8s single-node cluster that I don't think about at all really, I just have changes pushed to it directly with Pulumi (in a real environment I'd use something like ArgoCD) and everything just works nicely. Ingress to services is done through Cloudflare tunnels so I don't even have to port-forward or think about NAT or anything like this.
To update my personal site, I just do a git commit/push, the it's CI/CD builds builds a container and then updates the Pulumi config in the other repo to point to the latest hash, which then kicks off an action in my infra repo to do a Pulumi apply.
Currently it runs on Ubuntu but I'm thinking of using Talos (though it's still nice to be able to just SSH to the box and mess around with files).
I'm not sure why people struggle with this, or the benefits of this approach, so much? It seems like a lot of complexity if you're inexperienced, but if you've been working with computers for a long time, it isn't particularly difficult—there are far more complicated things that computers do.
I could throw the box (old macbook) in a lake and be up and running with every service on a new box in an hour or so. Or I could run it on the cloud. Or a VPS, or metal, or whatever really, it's a completely portable setup.
Just for my home server, I have more than 10 containers for home assistant, vpn, library management for movies/tv/music, photos backup, password manager, and a notes server. I started without knowing what docker was, and in less than a year realized running services directly on my OS was more hassle than I wanted both with compatibility between services dependencies, networking setup for them, and configuring reboots and upgrades. I would say the reproducibility and configurability is easily worth the slight overhead and in my experience even reduced it.
Suddenly you're in a team with 2-3 people and one of them likes to git push broken code and walk-off.
Okay, lets make this less about working with a jack-ass, same setup, but each 5 minutes of downtime cost you millions of dollars. One of your pushes work locally but don't work on the server.
The point of a more structed / complex CI/CD process is to eliminate failures. As the stakes become higher, and the stack becomes more complex, the need for the automation grows.
Docker is just a single part of that automation that makes other things / possible / lowers specific class of failures.
I know well what you are talking about since I did something similar, but I finally moved to Docker with Kamal (except one project I still have to move). The advantage of Docker's reproducibility is to have a peace of mind when comes to rollbacks and running exact versions due to system dependencies. If anyone is curious I wrote Kamal Handbook to help people adopt Kamal which I think brings all the niceness to Docker deployment so it's not annoying.
> Is the reproducibility of docker really worth the added overhead of managing containers, docker compose, and running daemons on your devbox 24/7?
Why wouldn't it be, containers are super easy to manage, dockerd uses bugger all resources in dev (on Linux anyway) and docker compose files are the simplest setup scripts I've ever used
In OpenWrt there is ujail, you give it an ELF (or multiple) to run, it'll parse them to find all the libraries they need, then it creates a tmpfs and mount bind read only the required files.
https://github.com/openwrt/procd/blob/dafdf98b03bfa6014cd94f...
I interviewed for a startup that does exactly this, except also for syscalls etc. They're mainly focused on security and not size. https://bifrostsec.com/
(I ended up taking another offer but I still think they're onto something.)
To provide 1 contrary opinion to all the others saying they have a problem:
Podman rocks for me!
I find docker hard to use and full of pitfalls and podman isn't any worse. On the plus side, any company I work for doesn't have to worry about licences. Win win!
> On the plus side, any company I work for doesn't have to worry about licences. Win win!
Was this a deal breaker for any company?
I ask because the Docker Desktop paid license requirement is quite reasonable. If you have less than 250 employees and make less than $10 million in annual revenue it's free.
If you have a dev team of 10 people and are extremely profitable to where you need licenses you'd end up paying $9 a year per developer for the license. So $90 / year for everyone, but if you have US developers your all-in payroll is probably going to be over $200,000 per developer or roughly $2 million dollars. In that context $90 is practically nothing. A single lunch for the dev team could cost almost double that.
To me that is a bargain, you're getting an officially supported tool that "just works" on all operating systems.
The problem isn’t generally the cost, it’s the complexity.
You end up having to track who has it installed. Hired 5 more people this week? How many of them will want docker desktop? Oh, we’ve maxed the licenses we bought? Time to re-open the procurement process and amend the purchase order.
This math sounds really simple until you work for a company that is “profitable” yet constantly turning over every sofa cushion for spare change. Whuch describes most publicly traded companies.
It can be quite difficult to get this kind of money for such a nominal tool that has a lot of free competition. Docker was very critical a few years ago, but “why not use podman or containerd or…” makes it harder to stand up for.
> If you have a dev team of 10 people and are extremely profitable to where you need licenses you'd end up paying $9 a year per developer for the license.
It doesn't quite change your argument, but where have you seen $9/year/dev?
The only way I see a $9 figure is the $9/month for Docker Pro with a yearly sub, so it's 12*$9=$108/year/dev or $1080/year for your 10 devs team.
Also it should be noted that Docker Pro is intended for individual professionals, so you don't have collaboration features on private repos and you have to manage each licence individually, which, even for only 10 licences, implies a big overhead.
If you want to work as a team you need to take the Docker Team licence, at $15/month/dev on a yearly sub, so now you are at $1800/year for your 10 devs team.
Twenty times more than your initial figure of $90/year.
Still, $1800 is not that much in the grand scheme of things, but then you still have to add a usual Atlassian sub, an Office365/GWorkspace sub, an AI sub...
You can end-up paying +$200/month/dev just in software licences, without counting the overhead of managing them.
It is at the company I currently work for. We moved to Rancher Desktop or Podman (individual choice, both are Apache licensed) and blocked Docker Desktop on IT's device management software. Much easier than going through finance and trying to keep up with licenses.
If your enterprise with a large engineering team that isn’t a software company, you are a cost center. So anything related to developer tools is rarely funded. It will mostly be - use the free stuff and suck it up.
Either that or you have a massive process to acquire said licenses with multiple reporting requirements. So, you manager doesn’t need the headache and says just use the free stuff and move on.
I used to use docker. I use podman now. Are there teams in my enterprise who have docker licenses - maybe. But tracking them down and dealing with the process of adding myself to that “list” isn’t worth the trouble.
I work for a $2 billion/yr company and we need three levels of approval for a Visio license. I've never been at a large corp where you could just order shit like that. You'll have to fill out forms , have a few meetings about it, business justification spreadsheets etc, then get told it's not in the budget.
> I ask because the Docker Desktop paid license requirement is quite reasonable. If you have less than 250 employees and make less than $10 million in annual revenue it's free.
It is for now, but I can't think of a player as large as Docker that hasn't pulled the rug out from under deals like this. And for good reason, that deal is probably a loss leader and if they want to continue they need to convert those free customers into paying.
The problem is not the cost. It's complexity. From a buyer perspective literally fighting with the procurement team is a nightmare.
And usually the need is coming from someone below C-level. So you have to: convince your manager and his manager
convince procurement team
it has to be in a budget (and usually it's much easier to convince to pay for the dinner)
than you have a procurement team
than you need to go through vendor review process (or at least chase execution)
This is reality in all big companies that this rule applies to. It's at least a quarter project.
Once I tried to buy a $5k/yr software license. The Sidekiq founder told me (after two months of back and forth) that he's done and I have to pay by CC (which I didn't had as miserable team lead).
> you'd end up paying $9 a year per developer for the license
It's only 9 bucks a year, its only 5 bucks a month, its less than a dollar a day.
Docker, ide, ticking system, GitHub, jira, sales force, email, office suit, Figma.... all of a sudden your spending 1000 bucks a month per staff member for a small 10 person office.
Meanwhile AWS is charging you .01xxxx for bandwidth, disk space, cpu time, s3 buckets, databases. All so tiencent based AI clients from China hammer your hardware and run up your bill....
Yep! What startup has the goal of making less than $10 million in annual revenue? That sentence was absolutely a deal breaker for the CEO and CTO of our last company.
We only run Podman Desktop if ever because for large companies it is cost prohibitive. We also found that most people don’t need *Desktop at all. Command line works fine
It's not the money, it's the bureaucracy. You can't just buy software, you need a justification, a review board meeting, marketplace survey with explanations of why this particular vendor was chosen over others with similar products, sign off from the management chain, yearly re-reviews for the support contract, etc...
And then you need to work with the vendor to do whatever licensing hoops they need to do to make the software work in an offline environment that will never see the Internet, something that more often than not blows the minds of smaller vendors these days. Half the time they only think in the cloud and situations like this seem like they come from Mars.
The actual cost of the product is almost nothing compared to the cost of justifying its purchase. It can be cheaper to hire a full time engineer to maintain the open source solutions just to avoid these headaches. But then of course you get pushback from someone in management that goes "we want a support contract and a paid vendor because that's best practices". You just can't win sometimes.
for reference a jp dev will be paid around $50,000. most of the world will probably be in the 10k-50k range except a few places (switzerland, luxembourg, usa?).
atlassian and google and okta and ghe and this and that (claude code?). that eventually starts to stack up.
$90 is also like 1.5 hours of work that I would've spent debugging podman anyway. And I've spent more than a few hours every time podman breaks, it to be honest.
Even large companies with thousands of developers have budgets to manage and often times when the CT/IO sees free as an option that's all that matters.
I've seen a weird thing on their service agreement:
Use Restrictions. Customer and its Users may not and may not allow any third party to:
[...]
10. Access the Service for the purpose of developing or operating products or services intended to be offered to third parties in competition with the Services[...]
Emphasis mine on 'operating'.
So I cannot use Docker Desktop to operate, for example: ECR, GCR or Harbor?
At my job going through procurement for something like Docker Desktop when there are free alternatives is not worth it.
It takes forever, so long that I'll forget that I asked for something. Then later when they do get around to it, they'll take up more of my time than it's worth on documentation, meetings, and other bullshit (well to me it's bullshit, I'm sure they have their reasons). Then when they are finally convinced that yes a Webstorm license is acceptable, they'll spend another inordinate amount of time trying to negotiate some deal with Jetbrains. Meanwhile I gave up 6 months ago and have been paying the $5 a month myself.
To bring up AI, and the eventual un-subsidizing of costs; if $9 a year is too much for docker... Then even the $20/mo (June) price tag is too high for AI, much less $200 (August), or $2000? (post subsidizing)
I think that I never saw somebody using Docker Desktop. I saw running containers with the command line everywhere, but I maybe I did not notice. No licenses for the command line tools, right?
It's a deal breaker because it was previously free to use, and frankly it's not worth $1 a month given there are better paid alternatives, let alone better free alternatives.
You need a compliance department and attorneys to look over licenses and agreements. It's a real hassle and not really related to cost of the license itself.
Yes. I worked for a company with a few thousand developers and we swapped away from Docker one week with almost no warning. It was a memorable experience.
It's not the cost, it's the headache. Do I need to worry about setting up SSO, do I need to work with procurement, do I need to do something in our SOC2 audit, do I need to get it approved as an allowed tool, etc.
Whether it's $100/year or $10k/year it's all the same headache. Yes, this is dumb, but it's how the process works at a lot of companies.
Whereas if it's a free tool that just magically goes away. Yes, this is also dumb.
None of your companies need to worry about licenses. Docker ENGINE is free and open source. Docker DESKTOP is a software suite that requires you to purchase a license to use in a company.
But Docker Engine, the core component which works on Linux, Mac and Windows through WSL2, that is completely and 1000% free to use.
>This section describes how to install Docker Engine on Linux, also known as Docker CE. Docker Engine is also available for Windows, macOS, and Linux, through Docker Desktop.
I'm not an expert but everything I read online says that Docker runs on Linux so with Mac you need a virtual environment like Docker Desktop, Colima, or Podman to run it.
Low-quality UX (e.g. you have to switch tabs and switch back if you ever want to see the current state of your containers, because it loads it once when you open the tab and never updates, and doesn't even give you a button to refresh it), lack of documentation, behavioural changes that happen silently (e.g. it autoupdates which changes the VM hostname, so the thing that was working yesterday doesn't work today and you have no idea why) and general flakiness.
I concur. My company is using Rancher Desktop on Windows machines. No problems. As long as you use don't care about GUI, and just use CLI dommands ("docker" , "docker compose" ).
On a few machines now ive had Podmans windows uninstaller fail to remove all its components and cause errors on start up due to postman not being found. Even manually removing leftover services and start up items didn't fix the issue. Its a constant source of annoyance.
It's works great until you need that one option from Docker Compose that is missing in Podman Compose (which is written in Python for whatever reason, yeah...).
I love podman, and, like others have said here, it does not always work with every container.
I often try to run something using podman, then find strange errors, then switch back to docker. Typically this is with some large container, like gitlab, which probably relies on the entirety of the history of docker and its quirks. When I build something myself, most of the time I can get it working under podman.
This situation where any random container does not work has forced me to spin up a VM under incus and run certain troublesome containers inside that. This isn't optimal, but keeps my sanity. I know incus now permits running docker containers and I wonder if you can swap in podman as a replacement. If I could run both at the same time, that would be magical and solve a lot of problems.
There definitely is no consistency regarding GPU access in the podman and docker commands and that is frustrating.
But, all in all, I would say I do prefer podman over docker and this article is worth reading. Rootless is a big deal.
I presume that the bulk of your issues are with container images that start their PID 1s as root. Podman is rootless by default, so this causes problems.
What you can do if you don't want to use Docker and don't want to maintain these images yourself is have two Podman machines running: one in rootful mode and another in rootless mode. You can, then, use the `--connection` global flag to specify the machine you want your container to run in. Podman can also create those VMs for you if you want it to (I use lima and spin them myself). I recommend using --capabilities to set limits on these containers namespaces out of caution.
Podman Desktop also installs a Docker compatibility layer to smooth over these incompatibilities.
> I love podman, and, like others have said here, it does not always work with every container.
Which is probably one of the motivations for the blog post. Compatibility will only be there once a large enough share of users use podman that it becomes something that is checked before publish.
Weird, we run GitLab server and runners all on podman. Honestly I wish we would switch to putting the runners in k8s. But it works well. We use Traefik.
Yeah, I had it running using podman, but then had some weird container restarts. I switched back to docker and those all went away. I am sure the solution is me learning more and troubleshooting podman, but I just didn't spend the time, and things are running well in an isolated VM under docker.
That's good to know it works well for you, because I would prefer not to use docker.
The main issue is podman support on Ubuntu.
Ubuntu ships outdated podman versions that don't work out of the box.
So I use podman v5, GitHub actions uses podman v3, and my coworkers on Ubuntu use docker.
So now my script must work with old podman, recent podman and docker
Additionally, there aren't even any trusted repos out there building/publishing a .deb for it. The ones that I could find when I searched last were all outdated or indicated they were not going to keep moving forward.
I could get over this. But, IMO, it lends itself to asking the "why" question. Why wouldn't Podman make installing it easier? And the only thing that makes sense to me is that RedHat doesn't want their dev effort supporting their competitor's products.
That's a perfectly reasonable stance, they owe me nothing. But, it does make me feel that anything not in the RH ecosystem is going to be treated as a second-class citizen. That concerns me more than having to build my own debs.
They publish statically-linked binaries on GitHub [0], so to install it, you just need to download and unpack a single file. But you don't get any automatic updates like you would if they provided an apt repository.
I don't know about this specific case but Debian and or Ubuntu having outdated software is a common Debian/Ubuntu problem which nearly always is cause by Debian/Ubuntu itself (funnily if it's outdated in Ubuntu doesn't mean it's outdated in Debian and the other way around ;=) ).
Yeah, the lack of an official upstream .deb that is kept up to date (like the official Docker .deb repos) for Ubuntu really kills using podman for most of my internal use cases.
This is my biggest problem too, and it's not just my problem but Podman's problem. Lack of name recognition is big for sure compared to Docker, but to me this version mismatch problem is higher on the list and more sure to keep Podman niche. Distros like Ubuntu always ship with older versions of software, it's sadly up to the maintainer to release newer versions, and Podman just doesn't seem interested in doing that. I don't know if it was their goal but it got me to use some RedHat derivative on my home server just to get a later version.
LTS in general being slow to uptake new versions of software is a feature not a bug. It gives predictability at the cost of having to deal with older versions of software.
With Ubuntu at least, some upstreams publish official PPAs so that you aren't stuck on the rapidly aging versions that Canonical picks when they cut an LTS release.
Debian I found out recently has something similar now via "extrepo".
Since Podman is open source, Ubuntu (and others) are able to package it and and update it themselves if they choose. But I think it's understandable that Red Hat would not want to pay their development teams to package their software for a direct competitor.
Yes! I really like the idea of podman, but after 4 hours trying to make it work on 24.04, I reverted to Docker and compose.
There is some dissonance in presenting Podman as a plug-in replacement for Docker, and making it so damn hard to install on (some category's) most popular contemporary LTS Linux distro.
I had this problem on the latest Pop OS LTS (I know, I know). Took about 4 hours to find the magic incantation that would let me install podman >= 4 without breaking anything else.
That's an Ubuntu issue though, they ship lots of outdated software. Nginx, PHP, PostgreSQL, Podman, etc, the critical software that must be updated asap, even with stable versions they all require a PPA to be properly updated.
> If your Docker Compose workflow is overly complex, just convert it to Kubernetes YAML. We all use Kubernetes these days, so why even bother about this?
I find that kubernetes yaml are a lot more complex than docker compose. And while I do, no, not everybody uses kubernetes.
Use an LLM is not a solution. It's effectively telling you to switch your brain off and hope nothing goes wrong in the future. In reality things do go wrong and any conversation should be done with a good understanding of the system involved.
Both (K8s and Compose) are well defined schemas, hence the conversation is mere mapping via search & replace. Bunch of `sed` statements could do that, LLM is an overkill for the job.
Meanwhile, kompose.io exists, which is exactly what it does (but with Go templates as far as I can tell)
You don’t need an LLM for this. Use `kubectl` to create a simple pod/service/deployment/ingress/etc, run `kubectl get -o yaml > foo.yaml` to bring it back to your machine in yaml format, then edit the `foo.yaml` file in your favorite editor, adding the things you need for your service, and removing the things you don’t, or things that are automatically generated.
As others have said, depending on an LLM for this is a disaster because you don’t engage your brain with the manifest, so you aren’t immediately or at least subconsciously aware of what is in that manifest, for good or for ill. This is how bad manifest configurations can drift into codebases and are persisted with cargo-cult coding.
I disagree with you on that. Kubernetes YAML is on the same level of complexity as docker compose and sometimes even easier.
But verbosity - yeah, kubernetes is absolutely super-verbose. Some 100-line docker-compose could easily end up as 20 yamls of 50 lines each. kubectl really needs some sugar to convert yamls from simple form to verbose and back.
We have about 30 services running in 4 environments, including dev. I desperately want a better kustomize that removes most of the boilerplate and adds linting (like, every process should have a ram limit, but no cpu limit). I estimate about 75% of the lines of YAML are redundant.
This is a conflation of "Simple" and "Easy" (rather, "complex" and "hard"). 'Simple vs Complex' is more or less objective, 'Easy vs Hard' is subjective, and changes based on the person.
And of course, Easy =/= Simple, nor the other way around.
The "podman generate systemd" command from the article is deprecated. The alternative are Podman Quadlets, which are similar to (docker-)compose.yaml, but defined in systemd unit files.
Which actually makes a lot of sense, to hand over the orchestration / composing to systemd, since it’s not client <> server API calls (like with docker) anymore but actual userland processes.
Yep. It works even better on a declarative distro like NixOS because you can define and extend your systemd services (including containers) from a single config.
Taking this further (self-plug), you can automatically map your Compose config into a NixOS config that runs your Compose project on systemd!
It totally does! On the con side, I find systemd unit files a lot less ergonomic to work with than compose files that can easily be git-tracked and colocated.
One challenge I have come across is mapping multi-UID containers to a single host user.
By default, root in the container maps to the user running the podman container on the host. Over the years, applications have adopted patterns where containers run as non-root users, for example www-data aka UID 33 (Debian) or just 1000. Those no longer map to your own user on the host, but subordinate IDs. I wish there was an easy way to just say "ALL container UIDs map to single host user". The uidmap and userns options did not work for me (crun has failed executing those containers).
I don’t see the use case for mapping to subordinate IDs. It means those files are orphaned on the host and do not belong to anyone, when used via volume mapping?
If I understand things correctly, this is Linux namespaces limitation, so tools like Docker or Podman will not be able to support such mapping without support from Linux. But I'm afraid the requirement for UIDs to be mapped 1:1 is fundamental, otherwise, say two container users 1000 and 0 are mapped to the same host user 1000. Who then should be displayed in the container as the owner of a file that is owned by the user 1000 on a host?
A second challenge with the particular setup I’m trying is peer authentication with Postgres, running bare metal on the host. I mount the Unix socket into the container, and on the host Postgres sees the Podman user and permits access to the corresponding DB.
Works really well but only if the container user is root so maps natively. I ended up patching the container image which was the path of least resistance.
I've been dealing with setting up Podman for work over the last week or so, and I wouldn't wish that on my worst enemy.
If you use rootless Podman on a Redhat-derived distribution (which means Selinux), along with a non-root user in your container itself, you're in for a world of pain.
Either the machine is a single security domain, in which case running as root is no issue, or it's not and you need actual isolation in which case run VMs with Firecracker/Kata containers/etc.
Rootless is indeed a world of pain for dubious security promises.
One of the major use cases was multi-user HPC systems. Because they can be complicated, it’s not uncommon for bioinformatics data analysis programs to be distributed as containers. Large HPC clusters are multi-tennant by nature, so running these containers needs to be rootless.
There are existing tools that fill this gap (Singularity/Apptainer). But, there is always friction when you have to use a specialized tool versus the default. For me, this is a core usecase for rootless containers.
For the reduced feature set we need from containers in bioinformatics, rootless is pretty straightforward. You could get largely the same benefits from chroots.
Where I think the issues start is when you start to use networking, subuids, or other features that require root-level access. At this level, rootless because a tedious exercise in configuration that probably isn’t worth the effort. The problem is, the features I need will be different from the features you need. Satisfying all users in a secure was may not be worth it.
I see your point but I wouldn't let the perfect be the enemy of the good.
If I just want to run a random Docker container, I'm grateful I can get at least "some security" without paying as much in setup/debugging/performance.
Of course, ideally I wouldn't have to choose and the thing that runs the container would be able to run it perfectly securely without me having to know that. But I appreciate any movement in that direction, even if it's not perfect.
Rootless is nice because if you mount some directory in, all the files don't end up owned by root. You can get around that by custom building every image so the user has your user id, but that's a pain.
Sure. Constructing the case to shoot yourself in the foot is not a big problem. But in reality things mostly just work. I’m happily running a bunch of services behind a (nginx) reverse proxy as rootless containers. Forgejo, the forgejo runner to build stuff, uptime-kuma and more on a bunch of RHEL10 machines with SELinux enabled.
I have done the same. It's not too bad - just don't rely on LLMs to design your quadlets or systemd unit files. Read the docs for the exact podman version you use and it's pretty okay.
Although it would be podman's job to explain how to set up Selinux, to avoid issues with it (and not just "disable it" as the answer). That is, if they list themselves as available for OS with Selinux.
That surprises me too. Podman is spearheaded by Redhat and Fedora/RHEL was one of the earliest distros to adopt it and phase out docker. Why wouldn't they have the selinux config figured out?
We went through an org wide Docker -> Podman migration and it went _relatively_ smooth. Some hiccups along the way but nothing that the SysDev team couldn't overcome.
Meanwhile it works perfectly fine without any fuzz on my two Fedora Silverblue setups. This sounds less like a case of "Podman is suffering by definition" and more a case of a bunch of different variables coming together in a less than ideal way.
I've setup a few podman machines (on debian), and generally liked it. I've been struggling for 2 days now to get a k8s server up, but that's not giving my any joy. (It doesn't like our nftables setup)
My anecdote: I've been using rootless podman on Ubuntu in production environments in multiple organizations (both startup and enterprise) for years without encountering a single issue related to podman itself.
I'm sure what you wrote here is true but i cant fathom how. Maybe its a rh specific issue? (Like how ubuntu breaks rootless bwrap by default)
Can confirm, have been doing exactly what GP says is a world of pain with no problems as soon as I learned what `:z` and `:Z` do and why they might be needed.
TL;DR: lowercase if a file from the host is shared with a container or a volume is shared between multiple containers. Uppercase in the same scenario if you want the container to take an exclusive lock on the volumes/files (very unlikely).
As a huge fan of podman this is definitely one of my disappointments. In the event that you're still struggling with this, the answer is using a --user systemd quadlet. You'll need to use machinectl (machinectl shell <user>@.host) for systemd commands to work, and you'll want to enable linger for that user.
One thing which just occurred to me, maybe it's possible to have a [container] and a [service].user in a quadlet?
Readit News
My deployment code worked by running the software outside of the jail environment and monitoring the running processes using `ptrace` to see what files it was trying to open. The `ptrace` output generated a list of dependencies, which could then be copied to create a deployment package.
This worked brilliantly and kept our deployments small and immutable and somewhat immune to attack -- not that being attacked was a huge concern in 2001 as it is today. When Docker came along, I couldn't help but recall that early work and wonder whether anyone has done a similar thing to monitor file usage within Docker containers and trim them down to size after observing actual use.
We set up a git post receive hook which built static files and restarted httpd on a git receive. Deployment was just 'git push live master'.
While I've used Docker a lot since then, that remains the single easiest deployment I've ever had.
I genuinely don't understand what docker brings to the table. I mean, I get the value prop. But it's really not that hard to set up http on vanilla Ubuntu (or God forbid, OpenBSD) and not really have issues.
Is the reproducibility of docker really worth the added overhead of managing containers, docker compose, and running daemons on your devbox 24/7?
Sounds great if you're only running a single web server or whatever. My team builds a fairly complex system that's comprised of ~45 unique services. Those services are managed by different teams with slightly different language/library/etc needs and preferences. Before we containerized everything it was a nightmare keeping everything in sync and making sure different teams didn't step on each others dependencies. Some languages have good tooling to help here (e.g. Python virtual environments) but it's not so great if two services require a different version of Boost.
With Docker, each team is just responsible for making sure their own containers build and run. Use whatever you need to get your job done. Our containers get built in CI, so there is basically a zero percent chance I'll come in in the morning and not be able to run the latest head of develop because someone else's dev machine is slightly different from mine. And if it runs on my machine, I have very good confidence it will run on production.
For me, as an ex-ops, the value proposition is to be able to package a complex stack made of one or more db, several services and tools (ours and external), + describe the interface of these services with the system in a standard way (env vars + mounts points).
It massively simplify the onboarding experience, make updating the stack trivial, and also allow devs, ci and prod to run the same version of all the libraries and services.
Not having to regularly rebuild the whole dev environment because I need to work on one particular Python app once a quarter and its build chain reliably breaks other stuff? Priceless.
If you want to handle all your deployments the same way, you can basically only choose between Nix and containers. Unfortunately, containers are far more popular and have more tooling.
Yes. Everything on my box is ephemeral and can be deleted and recreated or put on another box with little-to-no thought. Infrastructure-as-code means my setup is immutable and self-documented.
It's a little more time to set up initially, but now I know exactly what is running.
I don't really understand the 24/7 comment, now that it is set up there's very very little maintenance. Sometimes an upgrade might go askew but that is rare.
Any change to it is recorded as a git commit, I don't have to worry about logging what I've done ever because it's done for me.
Changes are handled by a GitHub action, all I have to do to change what is running is commit a file, and the infra will update itself.
I don't use docker-compose, I use a low-overhead microk8s single-node cluster that I don't think about at all really, I just have changes pushed to it directly with Pulumi (in a real environment I'd use something like ArgoCD) and everything just works nicely. Ingress to services is done through Cloudflare tunnels so I don't even have to port-forward or think about NAT or anything like this.
To update my personal site, I just do a git commit/push, the it's CI/CD builds builds a container and then updates the Pulumi config in the other repo to point to the latest hash, which then kicks off an action in my infra repo to do a Pulumi apply.
Currently it runs on Ubuntu but I'm thinking of using Talos (though it's still nice to be able to just SSH to the box and mess around with files).
I'm not sure why people struggle with this, or the benefits of this approach, so much? It seems like a lot of complexity if you're inexperienced, but if you've been working with computers for a long time, it isn't particularly difficult—there are far more complicated things that computers do.
I could throw the box (old macbook) in a lake and be up and running with every service on a new box in an hour or so. Or I could run it on the cloud. Or a VPS, or metal, or whatever really, it's a completely portable setup.
I still do that for all my personal projects! One of the advantages of docker is that you don't have to rebuild the thing on each deployment target.
https://gist.github.com/bonzini/1abbbdec739e77503945a3605e0e...
Suddenly you're in a team with 2-3 people and one of them likes to git push broken code and walk-off.
Okay, lets make this less about working with a jack-ass, same setup, but each 5 minutes of downtime cost you millions of dollars. One of your pushes work locally but don't work on the server.
The point of a more structed / complex CI/CD process is to eliminate failures. As the stakes become higher, and the stack becomes more complex, the need for the automation grows.
Docker is just a single part of that automation that makes other things / possible / lowers specific class of failures.
I find it easier to have the same interface for everything, where I can easily swap around ports.
Why wouldn't it be, containers are super easy to manage, dockerd uses bugger all resources in dev (on Linux anyway) and docker compose files are the simplest setup scripts I've ever used
I like docker because it's easy and I'm lazy
What exactly is your problem with OpenBSD? Shaming it completely out of context is kinda mean - they're the upstream for OpenSSH and LibreSSL.
> I genuinely don't understand what docker brings to the table.
I think you invalidated your own opinion here
(I ended up taking another offer but I still think they're onto something.)
Deleted Comment
Podman rocks for me!
I find docker hard to use and full of pitfalls and podman isn't any worse. On the plus side, any company I work for doesn't have to worry about licences. Win win!
Was this a deal breaker for any company?
I ask because the Docker Desktop paid license requirement is quite reasonable. If you have less than 250 employees and make less than $10 million in annual revenue it's free.
If you have a dev team of 10 people and are extremely profitable to where you need licenses you'd end up paying $9 a year per developer for the license. So $90 / year for everyone, but if you have US developers your all-in payroll is probably going to be over $200,000 per developer or roughly $2 million dollars. In that context $90 is practically nothing. A single lunch for the dev team could cost almost double that.
To me that is a bargain, you're getting an officially supported tool that "just works" on all operating systems.
Big companies are made of teams of teams.
The little teams don't really get to make purchasing decisions.
If there's a free alternative, little teams just have to suck it up and try to make it work.
---
Also consider that many of these expenses are born by the 'cost center' side of the house, that is, the people who don't make money for the company.
If you work in a cost center, the name of the game is saving money by cutting expenses.
If technology goes into the actual product, the cost for that is accounted for differently.
You end up having to track who has it installed. Hired 5 more people this week? How many of them will want docker desktop? Oh, we’ve maxed the licenses we bought? Time to re-open the procurement process and amend the purchase order.
It can be quite difficult to get this kind of money for such a nominal tool that has a lot of free competition. Docker was very critical a few years ago, but “why not use podman or containerd or…” makes it harder to stand up for.
It doesn't quite change your argument, but where have you seen $9/year/dev?
The only way I see a $9 figure is the $9/month for Docker Pro with a yearly sub, so it's 12*$9=$108/year/dev or $1080/year for your 10 devs team.
Also it should be noted that Docker Pro is intended for individual professionals, so you don't have collaboration features on private repos and you have to manage each licence individually, which, even for only 10 licences, implies a big overhead.
If you want to work as a team you need to take the Docker Team licence, at $15/month/dev on a yearly sub, so now you are at $1800/year for your 10 devs team.
Twenty times more than your initial figure of $90/year. Still, $1800 is not that much in the grand scheme of things, but then you still have to add a usual Atlassian sub, an Office365/GWorkspace sub, an AI sub... You can end-up paying +$200/month/dev just in software licences, without counting the overhead of managing them.
It is at the company I currently work for. We moved to Rancher Desktop or Podman (individual choice, both are Apache licensed) and blocked Docker Desktop on IT's device management software. Much easier than going through finance and trying to keep up with licenses.
Either that or you have a massive process to acquire said licenses with multiple reporting requirements. So, you manager doesn’t need the headache and says just use the free stuff and move on.
I used to use docker. I use podman now. Are there teams in my enterprise who have docker licenses - maybe. But tracking them down and dealing with the process of adding myself to that “list” isn’t worth the trouble.
It is for now, but I can't think of a player as large as Docker that hasn't pulled the rug out from under deals like this. And for good reason, that deal is probably a loss leader and if they want to continue they need to convert those free customers into paying.
And usually the need is coming from someone below C-level. So you have to: convince your manager and his manager convince procurement team it has to be in a budget (and usually it's much easier to convince to pay for the dinner) than you have a procurement team than you need to go through vendor review process (or at least chase execution)
This is reality in all big companies that this rule applies to. It's at least a quarter project.
Once I tried to buy a $5k/yr software license. The Sidekiq founder told me (after two months of back and forth) that he's done and I have to pay by CC (which I didn't had as miserable team lead).
It's only 9 bucks a year, its only 5 bucks a month, its less than a dollar a day.
Docker, ide, ticking system, GitHub, jira, sales force, email, office suit, Figma.... all of a sudden your spending 1000 bucks a month per staff member for a small 10 person office.
Meanwhile AWS is charging you .01xxxx for bandwidth, disk space, cpu time, s3 buckets, databases. All so tiencent based AI clients from China hammer your hardware and run up your bill....
The rent seeking has gotten out of hand.
And since when has Docker Desktop "just worked"?
It's not the money, it's the bureaucracy. You can't just buy software, you need a justification, a review board meeting, marketplace survey with explanations of why this particular vendor was chosen over others with similar products, sign off from the management chain, yearly re-reviews for the support contract, etc...
And then you need to work with the vendor to do whatever licensing hoops they need to do to make the software work in an offline environment that will never see the Internet, something that more often than not blows the minds of smaller vendors these days. Half the time they only think in the cloud and situations like this seem like they come from Mars.
The actual cost of the product is almost nothing compared to the cost of justifying its purchase. It can be cheaper to hire a full time engineer to maintain the open source solutions just to avoid these headaches. But then of course you get pushback from someone in management that goes "we want a support contract and a paid vendor because that's best practices". You just can't win sometimes.
Most Mac users I see using it struggle to see the difference between "image" and "container". Complete lack of understanding.
All the same stuff can easily be done from cli.
atlassian and google and okta and ghe and this and that (claude code?). that eventually starts to stack up.
Use Restrictions. Customer and its Users may not and may not allow any third party to: [...] 10. Access the Service for the purpose of developing or operating products or services intended to be offered to third parties in competition with the Services[...]
Emphasis mine on 'operating'.
So I cannot use Docker Desktop to operate, for example: ECR, GCR or Harbor?
It takes forever, so long that I'll forget that I asked for something. Then later when they do get around to it, they'll take up more of my time than it's worth on documentation, meetings, and other bullshit (well to me it's bullshit, I'm sure they have their reasons). Then when they are finally convinced that yes a Webstorm license is acceptable, they'll spend another inordinate amount of time trying to negotiate some deal with Jetbrains. Meanwhile I gave up 6 months ago and have been paying the $5 a month myself.
Correction: Docker Desktop is $9/month (not $9/year).
Whether it's $100/year or $10k/year it's all the same headache. Yes, this is dumb, but it's how the process works at a lot of companies.
Whereas if it's a free tool that just magically goes away. Yes, this is also dumb.
It's $9 per month not year.
It's not the price, it's that there is one. 1 penny would be too much because it prevents compose-ability of dev workstations.
But Docker Engine, the core component which works on Linux, Mac and Windows through WSL2, that is completely and 1000% free to use.
>This section describes how to install Docker Engine on Linux, also known as Docker CE. Docker Engine is also available for Windows, macOS, and Linux, through Docker Desktop.
https://docs.docker.com/engine/install/
I'm not an expert but everything I read online says that Docker runs on Linux so with Mac you need a virtual environment like Docker Desktop, Colima, or Podman to run it.
I have been using it by years. Tested it in Win11 and Linux Mint. I can have even a local kubernetes.
Deleted Comment
I often try to run something using podman, then find strange errors, then switch back to docker. Typically this is with some large container, like gitlab, which probably relies on the entirety of the history of docker and its quirks. When I build something myself, most of the time I can get it working under podman.
This situation where any random container does not work has forced me to spin up a VM under incus and run certain troublesome containers inside that. This isn't optimal, but keeps my sanity. I know incus now permits running docker containers and I wonder if you can swap in podman as a replacement. If I could run both at the same time, that would be magical and solve a lot of problems.
There definitely is no consistency regarding GPU access in the podman and docker commands and that is frustrating.
But, all in all, I would say I do prefer podman over docker and this article is worth reading. Rootless is a big deal.
What you can do if you don't want to use Docker and don't want to maintain these images yourself is have two Podman machines running: one in rootful mode and another in rootless mode. You can, then, use the `--connection` global flag to specify the machine you want your container to run in. Podman can also create those VMs for you if you want it to (I use lima and spin them myself). I recommend using --capabilities to set limits on these containers namespaces out of caution.
Podman Desktop also installs a Docker compatibility layer to smooth over these incompatibilities.
Which is probably one of the motivations for the blog post. Compatibility will only be there once a large enough share of users use podman that it becomes something that is checked before publish.
That's good to know it works well for you, because I would prefer not to use docker.
Deleted Comment
I could get over this. But, IMO, it lends itself to asking the "why" question. Why wouldn't Podman make installing it easier? And the only thing that makes sense to me is that RedHat doesn't want their dev effort supporting their competitor's products.
That's a perfectly reasonable stance, they owe me nothing. But, it does make me feel that anything not in the RH ecosystem is going to be treated as a second-class citizen. That concerns me more than having to build my own debs.
[0]: https://github.com/containers/podman/releases
What else can they do then having a package for every distro?
https://podman.io/docs/installation#installing-on-linux
Including instructions to build from source (including for Debian and Ubuntu):
https://podman.io/docs/installation#building-from-source
I don't know about this specific case but Debian and or Ubuntu having outdated software is a common Debian/Ubuntu problem which nearly always is cause by Debian/Ubuntu itself (funnily if it's outdated in Ubuntu doesn't mean it's outdated in Debian and the other way around ;=) ).
https://formulae.brew.sh/formula/podman
Yes I could use flatpack on ubuntu, however I feel like this is partly something Ubuntu/Debian should provide out-of-the-box
With Ubuntu at least, some upstreams publish official PPAs so that you aren't stuck on the rapidly aging versions that Canonical picks when they cut an LTS release.
Debian I found out recently has something similar now via "extrepo".
There is some dissonance in presenting Podman as a plug-in replacement for Docker, and making it so damn hard to install on (some category's) most popular contemporary LTS Linux distro.
I find that kubernetes yaml are a lot more complex than docker compose. And while I do, no, not everybody uses kubernetes.
On another note, podman can generate k8s yaml for you, which is a nice touch and easy way to transition.
Meanwhile, kompose.io exists, which is exactly what it does (but with Go templates as far as I can tell)
As others have said, depending on an LLM for this is a disaster because you don’t engage your brain with the manifest, so you aren’t immediately or at least subconsciously aware of what is in that manifest, for good or for ill. This is how bad manifest configurations can drift into codebases and are persisted with cargo-cult coding.
[edit: edit]
But verbosity - yeah, kubernetes is absolutely super-verbose. Some 100-line docker-compose could easily end up as 20 yamls of 50 lines each. kubectl really needs some sugar to convert yamls from simple form to verbose and back.
> Some 100-line docker-compose could easily end up as 20 yamls of 50 lines each.
Yeah, nothin to add here.
And of course, Easy =/= Simple, nor the other way around.
Taking this further (self-plug), you can automatically map your Compose config into a NixOS config that runs your Compose project on systemd!
https://github.com/aksiksi/compose2nix
Deleted Comment
By default, root in the container maps to the user running the podman container on the host. Over the years, applications have adopted patterns where containers run as non-root users, for example www-data aka UID 33 (Debian) or just 1000. Those no longer map to your own user on the host, but subordinate IDs. I wish there was an easy way to just say "ALL container UIDs map to single host user". The uidmap and userns options did not work for me (crun has failed executing those containers).
I don’t see the use case for mapping to subordinate IDs. It means those files are orphaned on the host and do not belong to anyone, when used via volume mapping?
A second challenge with the particular setup I’m trying is peer authentication with Postgres, running bare metal on the host. I mount the Unix socket into the container, and on the host Postgres sees the Podman user and permits access to the corresponding DB.
Works really well but only if the container user is root so maps natively. I ended up patching the container image which was the path of least resistance.
If you use rootless Podman on a Redhat-derived distribution (which means Selinux), along with a non-root user in your container itself, you're in for a world of pain.
Either the machine is a single security domain, in which case running as root is no issue, or it's not and you need actual isolation in which case run VMs with Firecracker/Kata containers/etc.
Rootless is indeed a world of pain for dubious security promises.
There are existing tools that fill this gap (Singularity/Apptainer). But, there is always friction when you have to use a specialized tool versus the default. For me, this is a core usecase for rootless containers.
For the reduced feature set we need from containers in bioinformatics, rootless is pretty straightforward. You could get largely the same benefits from chroots.
Where I think the issues start is when you start to use networking, subuids, or other features that require root-level access. At this level, rootless because a tedious exercise in configuration that probably isn’t worth the effort. The problem is, the features I need will be different from the features you need. Satisfying all users in a secure was may not be worth it.
If I just want to run a random Docker container, I'm grateful I can get at least "some security" without paying as much in setup/debugging/performance.
Of course, ideally I wouldn't have to choose and the thing that runs the container would be able to run it perfectly securely without me having to know that. But I appreciate any movement in that direction, even if it's not perfect.
I'm sure what you wrote here is true but i cant fathom how. Maybe its a rh specific issue? (Like how ubuntu breaks rootless bwrap by default)
A good reference answer: https://unix.stackexchange.com/questions/651198/podman-volum...
TL;DR: lowercase if a file from the host is shared with a container or a volume is shared between multiple containers. Uppercase in the same scenario if you want the container to take an exclusive lock on the volumes/files (very unlikely).
They are many different workarounds but it’s a known pain point.
One thing which just occurred to me, maybe it's possible to have a [container] and a [service].user in a quadlet?
Must not be a good sysadmin then. SELinux improves the security and software like podman can be relatively easily be made to work with it.
I use podman on my Fedora Workstation with selinux set to enforce without issues