why would they think no enterprise version is a selling point? I can't learn stuff at work and apply it at home easily with this product. if anything proxmox needs a more enterprise option with faster support and it would be a better product for me. The caveat is there needs to be a credible way to keep the opensource open and available, which proxmox has done so far.
Proxmox feels like a more apt comparison, as they both act like a controlplane for KVM virtual-machines and LXC containers across one or multiple hosts.
If you are interested in running kubernetes on top of incus, that is your kubernetes cluster nodes will be made up of KVM or LXC instances - I highly recommend the cluster-api provider incus https://github.com/lxc/cluster-api-provider-incus
This provider is really well done and maintained, including ClusterClass support and array of pre-built machine images for both KVM and LXC. It also supports pivoting the mgmt cluster on to a workload cluster, enabling the mgmt cluster to upgrade itself which is really cool.
Doesn't Vagrant spin up full VMs? Incus/LXD/LXC is about system containers. So like Docker, but with a full distro including init system running inside the container. They are much faster to spin up and have the best resource sharing possible.
Vagrant is not the right comparison against Incus for this use case. Vagrant is used to spin up VM or system container instances that are configured for software development and testing. But vagrant doesn't create those VMs or containers by itself. Instead, it depends on virtualization/container providers like VMware, Virtualbox, libvirt or lxc. In fact, you could create a provider plugin for vagrant to use Incus as its container/VM back end. (I couldn't find any, though.)
Instead of ephemeral containers, you have instances that are like VM (and incus can manage VM via qemu), so pretty much everything you would use a VM for, but if you do not need the kernel separation. It's more similar to FreeBSD jails than to docker.
if your question is genuine, then the simple answer would be full system inside container of LXD/Incus (I prefer to use term VE for Virtual Environment to distinguish from barebone containers of Docker style).
The full system, means, all your standard stuff works in expected way - crons, systemd units, sshd, multiple users, your regular backup solutions and so on. As well, that "system" can be dumped/exported/backuped/snapshotted as a whole - very much similar like you do with your vSphere/Qemu or whatever you use in your datacenters as hypervisor.
Foreseeing the question - yep, you can run Docker inside LXD/Incus VEs. In practical terms, that makes much simple when you need to give some dev team (who are of course not anywhere known to do sane things) access to environment with Docker access (which in 99% cases means that host's root level access is exposed).
Use cases are almost the same as Proxmox. You can orchestrate system containers or VMs. Proxmox runs lxc container images, while incus is built on top of lxc and has its own container images.
System vs Application containers: Both share virtualized kernels. Application containers usually run only a single application like web app (eg: OCI containers). System containers are more like VMs with systemd and multiple apps managed by it. Note: This differentiation is often ambiguous.
> Is this for people who want to run their own cloud provider, or that need to manage the infrastructure of org-owned VM's?
Yes, you could build a private cloud with it.
> When would you use this over k8s or serverless container runtimes like Fargate/Cloudrun?
You would use it when you need traditional application setups inside VMs or system containers, instead of containerized microservices.
I actually use Incus containers as host nodes for testing full fledged multinode K8s setups.
I know some webhosting provider that used one VM for every user. Now they moved to using this. Firstly low resource usage. If one uses ZFS or btrfs then one can save storage as in common bits are not duplicated across system containers. Note this is a system container - not traditional container. This one can be rebooted to get the previous state. It is not ephemeral.
System container tech like Incus powers efficient Linux virtualization environments for developers, so that you can have only one VM but many "machines". OrbStack machines on macOS work like this, and the way WSL works is similar (one VM and one Linux kernel, many guests sharing that kernel via system containers).
Just in case - I'm using LXD inside my WSL and it working great. BTRFS backed storage via loopfile, saves $$$.
For others, why it may be useful in regular sysadmin job:
* say doing Ansible scripting against LOCAL network is hell amount of time faster than against 300+ ms remote machines
* note that because you can use VE snapshots, it's very easy to ensure your playbook works fine without guessing what have you modified when testing things - just do rollback to "clean" state and start over
* creating test MariaDB 3 nodes cluster - easy peasy
* multiple distros available - need to debug Haproxy from say Rocky 8 linux? Check!
How do you handle updating the machine that Incus itself runs on? I imagine you have to be super careful not to introduce any breakage, because then all the VMs/containers go down.
What about kernel updates that require reboots? I have heard of ksplice/kexec, but I have never seen them used anywhere.
I'm not quite sure what's your question here? Very much similar to any other system which needs to reboot and you getting ready to these reboots in advance.
To some extent, of course things like vSphere/Virtuozzo and even LXD/Incus, and even simple Qemu/Virsh systems can do live migration of VMs, so you may care less on preparing things inside VMs to be fault taulerant, but to some extent.
I.e. if your team do run PostgreSQL, your run it in cluster with Patroni and VIPs and all that lovely industry standard magic and tell dev teams to use that VIP as entry point (in reality things bit more complicated with Haproxy/Pgbouncer on top, but enough to express the idea).
Readit News
https://lwn.net/Articles/940684/
[0]: https://tadeubento.com/2024/replace-proxmox-with-incus-lxd/
If you are interested in running kubernetes on top of incus, that is your kubernetes cluster nodes will be made up of KVM or LXC instances - I highly recommend the cluster-api provider incus https://github.com/lxc/cluster-api-provider-incus
This provider is really well done and maintained, including ClusterClass support and array of pre-built machine images for both KVM and LXC. It also supports pivoting the mgmt cluster on to a workload cluster, enabling the mgmt cluster to upgrade itself which is really cool.
I was surprised to come across this provider by chance as for some reason it's not listed on the CAPI documentation provider list https://cluster-api.sigs.k8s.io/reference/providers
Deleted Comment
The full system, means, all your standard stuff works in expected way - crons, systemd units, sshd, multiple users, your regular backup solutions and so on. As well, that "system" can be dumped/exported/backuped/snapshotted as a whole - very much similar like you do with your vSphere/Qemu or whatever you use in your datacenters as hypervisor.
Foreseeing the question - yep, you can run Docker inside LXD/Incus VEs. In practical terms, that makes much simple when you need to give some dev team (who are of course not anywhere known to do sane things) access to environment with Docker access (which in 99% cases means that host's root level access is exposed).
LXC containers used in incus run their own init, they act more like a VM.
However incus can also execute actual VMs via libvirt and since recently even OCI containers like docker.
Is this for people who want to run their own cloud provider, or that need to manage the infrastructure of org-owned VM's?
When would you use this over k8s or serverless container runtimes like Fargate/Cloudrun?
Use cases are almost the same as Proxmox. You can orchestrate system containers or VMs. Proxmox runs lxc container images, while incus is built on top of lxc and has its own container images.
System vs Application containers: Both share virtualized kernels. Application containers usually run only a single application like web app (eg: OCI containers). System containers are more like VMs with systemd and multiple apps managed by it. Note: This differentiation is often ambiguous.
> Is this for people who want to run their own cloud provider, or that need to manage the infrastructure of org-owned VM's?
Yes, you could build a private cloud with it.
> When would you use this over k8s or serverless container runtimes like Fargate/Cloudrun?
You would use it when you need traditional application setups inside VMs or system containers, instead of containerized microservices.
I actually use Incus containers as host nodes for testing full fledged multinode K8s setups.
For others, why it may be useful in regular sysadmin job:
* say doing Ansible scripting against LOCAL network is hell amount of time faster than against 300+ ms remote machines
* creating test MariaDB 3 nodes cluster - easy peasy* multiple distros available - need to debug Haproxy from say Rocky 8 linux? Check!
I personally use it mostly for deploying a bunch of system containers and some OCI containers.
But anyone who uses LXC, LXD, docker, libvirt, qemu etc. could potentially be interested in Incus.
Incus is just an LXD fork btw, developed by Stephane Graber.
What about kernel updates that require reboots? I have heard of ksplice/kexec, but I have never seen them used anywhere.
To some extent, of course things like vSphere/Virtuozzo and even LXD/Incus, and even simple Qemu/Virsh systems can do live migration of VMs, so you may care less on preparing things inside VMs to be fault taulerant, but to some extent.
I.e. if your team do run PostgreSQL, your run it in cluster with Patroni and VIPs and all that lovely industry standard magic and tell dev teams to use that VIP as entry point (in reality things bit more complicated with Haproxy/Pgbouncer on top, but enough to express the idea).