We tried to use this on our compute cluster for silicon design/verification. We gave up in the end and just went with the traditional TCL (now Lua) modules.
The problems are:
1. You can't have apptainers that use each other. The most common case was things like Make, GCC, Git etc. If Make is in a different apptainer to GCC then it won't work because as soon as you go into Make then it can't see GCC any more.
2. It doesn't work if any of your output artefacts depend on things inside the container. For example you use your GCC apptainer to compile a program. It appears to work, but when you run it you find it actually linked to something in the apptainer that isn't visible any more. This is also a problem for C headers.
3. We had constant issues with PATH getting messed up so you can't see things outside the apptainer that should have been available.
All in all it was a nice idea but ended up causing way more hassle than it was worth. It was much easier just to use an old OS (RHEL8) and get everything to work directly on that.
I think of using Apptainer/Singularity as more like Docker than anything else (without the full networking configs). These are all issues with traditional Docker containers as well, so I’m not sure how you were using the containers or what you were expecting.
For my workflows on HPC, I use apptainers as basically drop-in replacements for Docker, and for that, they work quite well. These biggest benefit is that the containers are unprivileged. This means you can’t do a lot of things (in particular complex networking), but it also makes it much more secure for multi-tenant systems (like HPC).
(I know Docker and Apptainer are slightly different beasts, but I’m speaking in broad strokes in a general sense without extra permissions).
You can also run Docker itself in rootless mode[1]. And if for some reason you don't want to run Docker, you can also use Podman or Incus instead, and they both support Docker images, as well as running unpriviliged. Finally, there's also Flox[2], which is a Nix-based application Sandbox, that I believe would align more towards your (and OP's) use case (unless you specifically require Docker image compatibility).
So unfortunately your example doesn't illustrate why Apptainer is a better option.
You don't mix and match pieces of containers, just like you wouldn't mix and match binaries from different distributions of Linux.
You can use a container as a single environment in which to do development, and that works fine. But they are by definition an isolated environment with different dependencies than other containers. The result of compiling something in a container necessarily needs to end up in its own container.
...that said, you could use the exact same container base image, and make many different container images from it, and those files would be compatible (assuming you shipped all needed dependencies).
> you wouldn't mix and match binaries from different distributions of Linux.
You can absolutely mix and match lots of different binaries from different sources on one Linux system. That's exactly what we're doing now with TCL modules.
> and make many different container images from it
Well yes, that's the problem. You end up either putting everything in one container (in which case why bother with a container?), or with a combinatorial explosion of every piece and version of software you might use.
TCL modules are better. They don't let you cheat like containers do, but in return you get a better system.
We put all of our software in a module and then we have a script for our projects that loads the appropriate software. I'm not sure I understand the question...
> Want single-file distribution model (although doesn't support deltas)
You can achieve that with docker by `docker save image-name > image-name.tar.gz` and `docker load --input image-name.tar.gz`.
It likewise doesn't support deltas but there was a link here on HN recently to something called "unregistry" which allows for doing "docker push" to deploy an image to a remote machine without a registry, and that thing does take deltas into account.
OCI image repositories are pretty ubiquitous nowadays and are trivial to setup.
I am sure that a lot of people have them deployed and don't even realize it. If you are using Gitea, Gitlab, Github, or any of their major forks/variations of you probably already have a place to put your images.
So I really don't know what the advantage of 'single file distribution model' is here.
This is probably why people don't bother sharing tarballs of docker images with one another even though it is has been a option this entire time.
> (this is an anti-pattern in other container technologies)
It is? I have no issues packing my development containers full of concurrent running processes. systemd even supports running as a "container init" out of the box, so you can get something that looks very similar to a full VM.
Apptainer and singularity ce are quite common in HPC. While both implementations fork the old singularity project, they are not really identical.
We use singularity in the HPCs (like Leonardo, LUMI, Fugaku, NeSI NZ, Levante) but some devs and researchers have apptainer installed locally.
We found a timezone bug a few days ago in our Python code (matplotlib,xarray,etc.), but that didn't happen with apptainer.
As the code bases are still a bit similar, I could confirm apptainer fixed it but singularity ce was still affected by the bug -- singularity replaces the UTC timezone file by the user's timezone, Helsinki EEST in our case in LUMI HPC.
> Apptainer and singularity ce are quite common in HPC. While both implementations fork the old singularity project, they are not really identical.
Apptainer is not a fork of the old Singularity project: Apptainer is the original project, but the community voted to change its name. It also came under the umbrella of the Linux Foundation:
If you ever use a shared cluster at a university or run by the government, Apptainer will be available, and Podman / Docker likely won't be.
In these environments, it is best not to use containers at all, and instead get to know your sysadmin and understand how he expects the cluster to be used.
docker and podman expect to extract images to disk, then use fancy features like overlayfs, which doesn't work on network filesystems -- and in hpc, most filesystems users can write to persistently are network filesystems.
apptainer images are straight filesystem images with no overlayfs or storage driver magic happening -- just a straight loop mount of a disk image.
this means your container images can now live on your network filesystem.
Maybe the idea is that flatpak can have better sandbox control over applications running in flatpak using xdg-dbus ie. you can select the permissions that you want to give to a flatpak application and so sometimes it can act near native and not be completely isolated like containers.
Also I am not sure if apptainers are completely isolated.
Though I suppose through tools like https://containertoolbx.org/ such point also becomes moot & then I guess if they move to container, doesn't it sort of become like toolbx?
To be honest, I think a lot of tools can have a huge overlap b/w them and I guess that's okay too
Containers in Linux are more a conceptual collection of different isolation techniques. Mostly just based on Linux namespaces. But things like cgroups, Linux capabilities, occasionally MAC (selinux, etc) and a few other items often get thrown in the mix.
After a quick view of the apptainer documentation it looks like it minimally takes advantage of user and mount namespaces. So each apptainer gets its own idea of what the users/groups are and what the file system looks like.
Flatpak is more about desktop application sandboxing. So while it does use user and mount namespaces like apptainer it takes advantage of more Linux features then that to help enhance the isolation.
Which appears to be the opposite of the point of apptainer. Apptainer wants to use containers that integrate tightly with the rest of the system with very little isolation versus Flatpak wants to be maximally isolated with only the permissions necessary for the application.
That isn't to say that apptainer can't use more Linux features to increase isolation. It supports the use of cgroups for resource quotas and can take advantage of different types of namespaces for network isolation among other things.
Now as far as "OSTree vs containers" statement you are replying to... This is kinda misleading.
OSTree is designed to manage binaries files in a way similar to git with text file. It isn't a type of container technology in itself. It just used for managing how objects on the file system are arranged and managed.
It is used by some flatpak applications, but it is used for things besides flatpak.
The 'containers' he mentioned is really a reference to OCI container image format.
OCI container images is, again, a way to manage the file system contents typically used in containers. It isn't a container technology itself.
It is like a tarball, but for file system images.
OCI containers is a standardized version of Docker images.
Due to the popularity and ubiquity of OCI image related tools and hosting software it makes sense for Flatpak to support it.
OCI images, when combined with bootc, also can be used to deploy Linux container images to "bare hardware". Which is gaining popularity in helping to create and deploy "immutable" or "atomic" Linux distributions. Fedora Atomic-based OSes seem to be moving to use Bootc with OCI over pure OSTree approach... although they still use OSTree in some capacity.
Incidentally Apptainer supports the use of OCI images (in addition to it's native SIF) as well as other commonly used container technologies like CNI. CNI is container network interface and is used with Kubernetes among other things.
Server - but this is kind of a wrong question. Apptainer is for running cli applications in immutable, rootless containers. The closest tool I can think of is Fedora Toolbx [1]. Apptainer is primarily used to distributing and reusing scientific computing tools b/c it doesn't allow root, doesn't allow for changes to the rootfs in each container, automatically mounts the working directory and works well with GPUs (that last point I can't personally attest to).
> Apptainer is for running cli applications in immutable, rootless containers.
What's the appeal of using this over unshare + chroot to a mounted tarball with a tmpfs union mount where needed? Saner default configuration? Saner interface to cgroups?
In my environment, the number one reason Apptainer is used has nothing to do with deployment, isolation, or software availability: it is to work around inode limits.
On our HPC cluster, each user has a quota of inodes on the shared filesystem. This makes installing some software with lots of files problematic (like Anaconda). An Apptainer image is a single file on the filesystem though (basically squashfs) so you can have those with as many files as you want in each.
Installing the same software normally is easy and works fine though, you just exchaust your quota.
Readit News
The problems are:
1. You can't have apptainers that use each other. The most common case was things like Make, GCC, Git etc. If Make is in a different apptainer to GCC then it won't work because as soon as you go into Make then it can't see GCC any more.
2. It doesn't work if any of your output artefacts depend on things inside the container. For example you use your GCC apptainer to compile a program. It appears to work, but when you run it you find it actually linked to something in the apptainer that isn't visible any more. This is also a problem for C headers.
3. We had constant issues with PATH getting messed up so you can't see things outside the apptainer that should have been available.
All in all it was a nice idea but ended up causing way more hassle than it was worth. It was much easier just to use an old OS (RHEL8) and get everything to work directly on that.
For my workflows on HPC, I use apptainers as basically drop-in replacements for Docker, and for that, they work quite well. These biggest benefit is that the containers are unprivileged. This means you can’t do a lot of things (in particular complex networking), but it also makes it much more secure for multi-tenant systems (like HPC).
(I know Docker and Apptainer are slightly different beasts, but I’m speaking in broad strokes in a general sense without extra permissions).
So unfortunately your example doesn't illustrate why Apptainer is a better option.
[1] https://docs.docker.com/engine/security/rootless/
[2] https://flox.dev/
Deleted Comment
You can use a container as a single environment in which to do development, and that works fine. But they are by definition an isolated environment with different dependencies than other containers. The result of compiling something in a container necessarily needs to end up in its own container.
...that said, you could use the exact same container base image, and make many different container images from it, and those files would be compatible (assuming you shipped all needed dependencies).
You can absolutely mix and match lots of different binaries from different sources on one Linux system. That's exactly what we're doing now with TCL modules.
> and make many different container images from it
Well yes, that's the problem. You end up either putting everything in one container (in which case why bother with a container?), or with a combinatorial explosion of every piece and version of software you might use.
TCL modules are better. They don't let you cheat like containers do, but in return you get a better system.
- Need to run more than one activity in a single container (this is an anti-pattern in other container technologies)
- HPC (and sometimes college) environments
- Want single-file distribution model (although doesn't support deltas)
- Cryptographically sign a SIF file without an external server
- Robust GPU support
You can achieve that with docker by `docker save image-name > image-name.tar.gz` and `docker load --input image-name.tar.gz`.
It likewise doesn't support deltas but there was a link here on HN recently to something called "unregistry" which allows for doing "docker push" to deploy an image to a remote machine without a registry, and that thing does take deltas into account.
I am sure that a lot of people have them deployed and don't even realize it. If you are using Gitea, Gitlab, Github, or any of their major forks/variations of you probably already have a place to put your images.
So I really don't know what the advantage of 'single file distribution model' is here.
This is probably why people don't bother sharing tarballs of docker images with one another even though it is has been a option this entire time.
It is? I have no issues packing my development containers full of concurrent running processes. systemd even supports running as a "container init" out of the box, so you can get something that looks very similar to a full VM.
We use singularity in the HPCs (like Leonardo, LUMI, Fugaku, NeSI NZ, Levante) but some devs and researchers have apptainer installed locally.
We found a timezone bug a few days ago in our Python code (matplotlib,xarray,etc.), but that didn't happen with apptainer.
As the code bases are still a bit similar, I could confirm apptainer fixed it but singularity ce was still affected by the bug -- singularity replaces the UTC timezone file by the user's timezone, Helsinki EEST in our case in LUMI HPC.
https://github.com/sylabs/singularity/issues/3686
Apptainer is not a fork of the old Singularity project: Apptainer is the original project, but the community voted to change its name. It also came under the umbrella of the Linux Foundation:
* https://apptainer.org/news/community-announcement-20211130/
Sylabs (where the original Singularity author first worked) was the one that forked off the original project.
https://journals.plos.org/plosone/article?id=10.1371/journal...
If you ever use a shared cluster at a university or run by the government, Apptainer will be available, and Podman / Docker likely won't be.
In these environments, it is best not to use containers at all, and instead get to know your sysadmin and understand how he expects the cluster to be used.
apptainer images are straight filesystem images with no overlayfs or storage driver magic happening -- just a straight loop mount of a disk image.
this means your container images can now live on your network filesystem.
Also I am not sure if apptainers are completely isolated.
Though I suppose through tools like https://containertoolbx.org/ such point also becomes moot & then I guess if they move to container, doesn't it sort of become like toolbx?
To be honest, I think a lot of tools can have a huge overlap b/w them and I guess that's okay too
https://www.redhat.com/en/blog/7-linux-namespaces
After a quick view of the apptainer documentation it looks like it minimally takes advantage of user and mount namespaces. So each apptainer gets its own idea of what the users/groups are and what the file system looks like.
Flatpak is more about desktop application sandboxing. So while it does use user and mount namespaces like apptainer it takes advantage of more Linux features then that to help enhance the isolation.
Which appears to be the opposite of the point of apptainer. Apptainer wants to use containers that integrate tightly with the rest of the system with very little isolation versus Flatpak wants to be maximally isolated with only the permissions necessary for the application.
That isn't to say that apptainer can't use more Linux features to increase isolation. It supports the use of cgroups for resource quotas and can take advantage of different types of namespaces for network isolation among other things.
Now as far as "OSTree vs containers" statement you are replying to... This is kinda misleading.
OSTree is designed to manage binaries files in a way similar to git with text file. It isn't a type of container technology in itself. It just used for managing how objects on the file system are arranged and managed.
It is used by some flatpak applications, but it is used for things besides flatpak.
The 'containers' he mentioned is really a reference to OCI container image format.
OCI container images is, again, a way to manage the file system contents typically used in containers. It isn't a container technology itself.
It is like a tarball, but for file system images.
OCI containers is a standardized version of Docker images.
Due to the popularity and ubiquity of OCI image related tools and hosting software it makes sense for Flatpak to support it.
OCI images, when combined with bootc, also can be used to deploy Linux container images to "bare hardware". Which is gaining popularity in helping to create and deploy "immutable" or "atomic" Linux distributions. Fedora Atomic-based OSes seem to be moving to use Bootc with OCI over pure OSTree approach... although they still use OSTree in some capacity.
Incidentally Apptainer supports the use of OCI images (in addition to it's native SIF) as well as other commonly used container technologies like CNI. CNI is container network interface and is used with Kubernetes among other things.
[1]: https://docs.fedoraproject.org/en-US/fedora-silverblue/toolb...
What's the appeal of using this over unshare + chroot to a mounted tarball with a tmpfs union mount where needed? Saner default configuration? Saner interface to cgroups?
On our HPC cluster, each user has a quota of inodes on the shared filesystem. This makes installing some software with lots of files problematic (like Anaconda). An Apptainer image is a single file on the filesystem though (basically squashfs) so you can have those with as many files as you want in each.
Installing the same software normally is easy and works fine though, you just exchaust your quota.