I wonder how hard it would be to port this over to a Karpenter variant. I suppose the Proxmox API won't do the same things as an AWS EC2 API so an operator-in-the-middle would still be required.
If your k8s nodes are the same height and width as your Proxmox servers, then yes. In my case, I have clusters of ~7 nodes, 100 cores, 1.75TB RAM, but my VMs tend to be much, much smaller. My web workers are each a handful of cores, I could double them a couple times before saturating the physical nodes core count.
I guess I just don't get why you'd have such small k8s nodes in a cluster, I have a cluster about half that size (in node and core count) but just run k8s directly on the metal. It means that things are quite elastic with an HPA + VPA on things that need it. I feel like we get better overall utilization of the hardware, but I'd be interested in hearing about some of the reasons for your approach.
Off the top of my head, there are probably a couple of 'pros' to doing it your way; one: experimentation is probably easier. Two, I'm willing to bet networking isn't anything too fancy (getting k8s to play nice on bare metal networks is a shitty thing, especially once multiple NICs are involved -- a LOT of networking stacks, ok, all of them, just 'assume' that the first network card it sees is THE network card it should use... and don't even bother to 'sort' them so its the same order every reboot).
> Thinly provisioned compute in the presence of other VMs is one reason
That's the only reason I can think of.
> I think this is aimed at people with multiple proxmox nodes in a cluster
I think so too, but then we're back to the original question again (i.e. why not just have kubernetes nodes instead of having proxmox nodes with kubernetes VMs on them).
Proxmox can also automatically provision zfs underneath, automatically set up ceph across nodes, has a nice ui for the firewall and more. Maybe the benefit is convenience.
I am a little tired of commenters lazily flinging this every time something in the infrastructure world comes up that they don't fully understand. Something doesn't become word salad just because you don't know what it means.
I might have problems understanding something -- say, "Comparison of implementations of in-order traversal of an AVL tree in Scala v/s Clojure"; but that doesn't mean it doesn't have a clear and intelligible meaning to someone who gets it.
In this case, I understand the idea here somewhat even without looking at the project. Kubernetes has very flexible auto-scaling capabilities. This seems to be be somehow allowing it to scale across a bunch of VMs hosted using Proxmox, which is an open-source VM hosting platform. There is no word salad about it.
I have heard of Proxmox but didn't really know what it did, so I went to Wikipedia, which tells me: "Proxmox Virtual Environment (Proxmox VE or PVE) is a hyper-converged infrastructure open-source software."
This is a little buzzwordy. I am also not sure that something can be "a software".
Readit News
Off the top of my head, there are probably a couple of 'pros' to doing it your way; one: experimentation is probably easier. Two, I'm willing to bet networking isn't anything too fancy (getting k8s to play nice on bare metal networks is a shitty thing, especially once multiple NICs are involved -- a LOT of networking stacks, ok, all of them, just 'assume' that the first network card it sees is THE network card it should use... and don't even bother to 'sort' them so its the same order every reboot).
That's the only reason I can think of.
> I think this is aimed at people with multiple proxmox nodes in a cluster
I think so too, but then we're back to the original question again (i.e. why not just have kubernetes nodes instead of having proxmox nodes with kubernetes VMs on them).
I might have problems understanding something -- say, "Comparison of implementations of in-order traversal of an AVL tree in Scala v/s Clojure"; but that doesn't mean it doesn't have a clear and intelligible meaning to someone who gets it.
In this case, I understand the idea here somewhat even without looking at the project. Kubernetes has very flexible auto-scaling capabilities. This seems to be be somehow allowing it to scale across a bunch of VMs hosted using Proxmox, which is an open-source VM hosting platform. There is no word salad about it.
This is a little buzzwordy. I am also not sure that something can be "a software".