I’ve been running an Incus cluster of 3 fairly beefy servers for about a year now. It’s my go-to recommendation for anyone wanting to setup a new virtualized environment.
One of my favorite features is how you can tag different cluster members for different architectures. In the same cluster, I can have traditional dual-socket x86 servers with a dozen DIMM slots as well as Raspberry Pis. The architecture tagging lets me strategize execution of ARM-based container workloads to be only on the Pis, or opt to run them via QEMU on the x86 platforms if that makes more sense in a particular scenario. Since I deal with a lot of embedded firmware, this offers a nice, flexible platform.
Stephen Graeber is also a long time contributor to the LXC project and his reasoning behind this fork and other changes are quite sound. I hope the project sees continued success. Stephen’s business model of offering consulting services for Incus systems also seems quite sound.
I used Proxmox for years to run a fairly comprehensive homelab, and a few months ago replaced the entire thing with Incus (on a debian host, haven't tried IncusOS yet). Incus is amazing and it makes so many things so much easier compared to Proxmox.
One thing in particular is permissions in unprivileged containers. In Proxmox, you have to do a bunch of somewhat confusing ID mapping. In Incus, it's as simple as setting "shift=true".
Also the profile system in Incus is really powerful and allowed me to deduplicate a ton of config.
Interesting. I recommend Incus over other hypervisor-oriented OS distributions (Proxmox, etc.), for many reasons, but one minor reason is that you can install it on Debian or Arch or whatever Linux you like. It's a comprehensive management layer for VMs and Linux containers, with good CLI, web UI, cloning/moving/copying/migrating etc., but... why does that need to be an OS?
However, I'm confident that if that is what you want, this is probably fantastic — Incus, including the old LXD (which was mainly built by the same core developers, until Canonical behaved in ways they didn't like, and they hard-forked LXD to create Incus) has been one of my favorite open-source projects for several years.
Fantastic software, steady stream of reliable releases, helpful community... Incus is great.
It doesn't. You can still run Incus on other platforms of choice.
Sample size 1 here, but a big advantage of the 'full-stack' approach is things like network config, storage management, boot safety etc all work out of the box and you then get a single API (and nice client) for the whole machine. I get the benefits of cloud infra, like not having to care (too much) about sysadmin, from some hardware sitting in the corner.
Previously I would still need to be maintaining that base layer too. That still makes sense for some environments, but particularly for home I just want my lights and music to work, and be able to play.
AFAIK LXD is still opensource, as are most if not all products from Canonical. I think the fork is because LXD when it was moved to Canonical made the community uneasy because of the way that they would integrate with Ubuntu lifecycle and tooling.
Really excited to try this out. I have a fleet of containers on ubuntu + incus. Not only does this do ZFS optimization, I look forward having easy container optimized backup, live cluster migration (to a different machine without downtime) and so much more.
I use Proxmox on fat servers, but for homelab-like setup Incus OS seems more like a sweet spot
I was hoping for easy backup via zfs send as well, but turns out it’s not so easy atm.
IncusOS does not give you shell access, you have to figure out IncusOS ways to do things via their CLI/API. I haven’t found an easy way to do incremental backup of the whole system yet. You can backup individual instances/volumes via incus export (which seems to use zfs send under the hood), but not the whole thing.
I have mixed feelings about their decision not to give you shell access. Guess those who want flexibility can always just install Incus on top of any Linux they like, but it would be nice to have an escape hatch for when IncusOS gives you almost everything you want…
I've been using Incus containers (not VMs) for running tests against a "real" OS and it's been an absolute game changer for me. It's granted me the ability to simultaneously spin up-and-down a plethora of fresh OSes on my local dev machine, which I then use as testing targets for components of my codebase that require Docker or systemd. With traditional containers, it's tricky to mimic those capabilities as they would exist on a normal VM.
Because both my project and Incus are written in Go, orchestrating Incus resources in my test code has been pretty seamless. And with "ephemeral" containers, if things start to get out of hand, I just need to stop the container to clean it up. Much easier than a 2-step process like it usually is.
Looking forward to seeing what's to come in IncusOS!
Incus is very nice and super featured, but suffers from a few issues, namely unintuitive/hard onboarding and bad defaults, which makes giving access to people annoying, as it requires teaching them first and that they can't just make a vm with a few clicks immediately, limited authentication and user control options, like if no external auth users must exist on underlying system, and with limited but very strict auth options requires a full domain and no proxying, currently (might get fixed partially later).
And finally, it suffers from hardcore tracking upstream, ie canonical/lxd(-ui), meaning they won't really do any changes that lxd wouldn't do, and thus are slaved to them : (
Sorry what? Lxd is NOT incus upstream. Incus was forked from lxd specifically to allow divergence and the licenses mean changes rather flow in the other direction. Not that canonical considers incus “upstream” - they’re just divergent forks at this point.
It's a lot more than that. Clustering, storage drivers, networking, etc makes up a whole virtual machine manager. It never says it's a hypervisor, it's a VMM as outlined on it's github: "Powerful system container and virtual machine manager"
One of my favorite features is how you can tag different cluster members for different architectures. In the same cluster, I can have traditional dual-socket x86 servers with a dozen DIMM slots as well as Raspberry Pis. The architecture tagging lets me strategize execution of ARM-based container workloads to be only on the Pis, or opt to run them via QEMU on the x86 platforms if that makes more sense in a particular scenario. Since I deal with a lot of embedded firmware, this offers a nice, flexible platform.
Stephen Graeber is also a long time contributor to the LXC project and his reasoning behind this fork and other changes are quite sound. I hope the project sees continued success. Stephen’s business model of offering consulting services for Incus systems also seems quite sound.
One thing in particular is permissions in unprivileged containers. In Proxmox, you have to do a bunch of somewhat confusing ID mapping. In Incus, it's as simple as setting "shift=true".
Also the profile system in Incus is really powerful and allowed me to deduplicate a ton of config.
Loading comment...
Loading comment...
LXD containers also are unprivileged by default.
Loading comment...
Loading comment...
Loading comment...
Loading comment...
Loading comment...
However, I'm confident that if that is what you want, this is probably fantastic — Incus, including the old LXD (which was mainly built by the same core developers, until Canonical behaved in ways they didn't like, and they hard-forked LXD to create Incus) has been one of my favorite open-source projects for several years.
Fantastic software, steady stream of reliable releases, helpful community... Incus is great.
It doesn't. You can still run Incus on other platforms of choice.
Sample size 1 here, but a big advantage of the 'full-stack' approach is things like network config, storage management, boot safety etc all work out of the box and you then get a single API (and nice client) for the whole machine. I get the benefits of cloud infra, like not having to care (too much) about sysadmin, from some hardware sitting in the corner.
I can literally
and start hacking.Previously I would still need to be maintaining that base layer too. That still makes sense for some environments, but particularly for home I just want my lights and music to work, and be able to play.
Loading comment...
https://github.com/canonical/lxd it's AGPL-V3
Loading comment...
Loading comment...
Loading comment...
I use Proxmox on fat servers, but for homelab-like setup Incus OS seems more like a sweet spot
IncusOS does not give you shell access, you have to figure out IncusOS ways to do things via their CLI/API. I haven’t found an easy way to do incremental backup of the whole system yet. You can backup individual instances/volumes via incus export (which seems to use zfs send under the hood), but not the whole thing.
I have mixed feelings about their decision not to give you shell access. Guess those who want flexibility can always just install Incus on top of any Linux they like, but it would be nice to have an escape hatch for when IncusOS gives you almost everything you want…
Loading comment...
Loading comment...
Because both my project and Incus are written in Go, orchestrating Incus resources in my test code has been pretty seamless. And with "ephemeral" containers, if things start to get out of hand, I just need to stop the container to clean it up. Much easier than a 2-step process like it usually is.
Looking forward to seeing what's to come in IncusOS!
And finally, it suffers from hardcore tracking upstream, ie canonical/lxd(-ui), meaning they won't really do any changes that lxd wouldn't do, and thus are slaved to them : (
Loading comment...
But this is definitely neat. I've found Incus quite handy for development environments, and a good compliment to docker.
Loading comment...