Not gonna lie. I designed a system that looks exactly like this. However, the plugin system was an operator instead of annotating a service which is kinda cleaner but it still felt dirty. A better approach is having a service registry to track the versions, values and clusters where this has been pushed but that means you needed a different API or extend the k8s API and deal with the whole auth/authz system.
I think the biggest hurdle to implement a solution like this is your application MUST be a 12 factor app which sadly is not the case from what I've seen. Many devs hardcode values when shipping their code and containers which make it really difficult to add nodes to a DAG. Had this change, SDLC testing would be waaay easier.
In the same space, labeling of all services is a thing. As you are annotating it can be tracked but if you had different gateways or destination routes or other things that need to be tweaked at the mesh level, it could be daunting to track it. In my case, I ensure the whole mesh representation of a version could be tracked by a label so if you had a app=vote1,version=release-4,feature=shiny you could track the shiny components across the whole cluster.
Another hurdle is you are tied to a service mesh implementation. Istio is ok but it can be a beast. It also constraints your org to implement other networking alternatives which is something we wanted to explore.
Man this brilliant and goes straight to test it with my development workflow. Namespace isolation was always PITA,but full isolation would require a dedicated clusters so having a best of both solution really changes the game for lean and cost effective continues delivery.
As for value prop, maybe emphasize multi tenancy a bit more as this has the most cost saving potential at scale.
For sure, probably test it over the weekend. I've dealt with k8s virtualization before and as a result, my current system uses contextual auto configuration for all it's services which makes it very easy to lift it from one kubernetes context to another, or even a docker only environment as it's often the case with CI.
However my primary challenge is chronic cluster under utilization after having rewriten the bulk of the system in Rust. Therefore virtualizating the cluster makes the most sense. I think Google Cloud conducted a study that found that the bulk, like 75%, of their K8s customer over provision by a wide margin. So there is definitely a market for cluster virtualization and even more so for multi tenancy.
Reading the docs, I consistently get blank page with an error after less than 30s: "Application error: a client-side exception" it feels strange that a text document needs to be an Application. I feel so old.
How is this different from vcluster? In fact vcluster has no dependency on Istio which I think is a huge detriment to kardinal.
We use cilium so the istio requirement becomes a non starter for us.
Regarding how Kardinal compares with vcluster, both aim to provide multitenancy within a single Kubernetes cluster, but they approach it differently.
vcluster is designed to offer a full Kubernetes experience to each tenant, including their own API server, control plane, and other Kubernetes-level components. This makes vcluster an excellent choice if you need strong isolation or want to test Kubernetes-specific features like custom CRDs or operators.
Kardinal, on the other hand, focuses on application-level multitenancy rather than providing the full Kubernetes stack to each tenant. This makes Kardinal lighter-weight and more efficient if you're primarily concerned with deploying and testing application-level changes.
Tilt works as an in-place hot-reload tool for containers. However, because it performs in-place updates, it doesn't support running multiple parallel and isolated versions of your cluster side-by-side.
That said, the timing of your question is perfect since we are about to push a Tilt integration that allows you to create an isolation flow and use Tilt to hot-reload and update that flow in-place! :)
You're completely right, ofrzeta! Kardinal does manage Istio for you, so you don't need to learn or interact with Istio at all. In fact, Kardinal acts as a transpiler, taking Kubernetes manifests with simple Kardinal annotations and generating and applying the Kubernetes and Istio resources for you.
This looks interesting, but is there a reason the kardinal-cli deploys kardinal-manager into the default namespace rather than taking a --namespace argument (for example)? Personally I never run anything in the default namespace but I am probably an outlier.
Congratulations on shipping! I like how opinionated Kardinal is, which means it should work nicely for anyone who shares the same kind of infras vision as you.
It's also such an interesting moment for you folks to show up on HN, I just shipped the first preview version of my Kubernetes operator(https://github.com/pier-oliviert/sequencer) that manages ephemeral environments. I can see some things that are similar with both our options as well as some things that are quite different.
Maybe if I had one question is: What made you go for Istio as the main network mesh?
Nice work on Sequencer! I really liked the Workspace CRD.
Nothing too specific on choosing Istio, but it seemed like a popular and battle-tested option to start the implementation with. Kardinal is a fairly young project (about 2 months old), and we expect it to adopt more "service mesh backends" in the future.
Readit News
I think the biggest hurdle to implement a solution like this is your application MUST be a 12 factor app which sadly is not the case from what I've seen. Many devs hardcode values when shipping their code and containers which make it really difficult to add nodes to a DAG. Had this change, SDLC testing would be waaay easier.
In the same space, labeling of all services is a thing. As you are annotating it can be tracked but if you had different gateways or destination routes or other things that need to be tweaked at the mesh level, it could be daunting to track it. In my case, I ensure the whole mesh representation of a version could be tracked by a label so if you had a app=vote1,version=release-4,feature=shiny you could track the shiny components across the whole cluster.
Another hurdle is you are tied to a service mesh implementation. Istio is ok but it can be a beast. It also constraints your org to implement other networking alternatives which is something we wanted to explore.
I do like the project uses Nix =).
As for value prop, maybe emphasize multi tenancy a bit more as this has the most cost saving potential at scale.
However my primary challenge is chronic cluster under utilization after having rewriten the bulk of the system in Rust. Therefore virtualizating the cluster makes the most sense. I think Google Cloud conducted a study that found that the bulk, like 75%, of their K8s customer over provision by a wide margin. So there is definitely a market for cluster virtualization and even more so for multi tenancy.
Luckily the docs site is just generated from markdown files in the repo, so feel free to browse around the various `page.mdx` files here if you prefer that: https://github.com/kurtosis-tech/kardinal/tree/main/website/...
Regarding how Kardinal compares with vcluster, both aim to provide multitenancy within a single Kubernetes cluster, but they approach it differently.
vcluster is designed to offer a full Kubernetes experience to each tenant, including their own API server, control plane, and other Kubernetes-level components. This makes vcluster an excellent choice if you need strong isolation or want to test Kubernetes-specific features like custom CRDs or operators.
Kardinal, on the other hand, focuses on application-level multitenancy rather than providing the full Kubernetes stack to each tenant. This makes Kardinal lighter-weight and more efficient if you're primarily concerned with deploying and testing application-level changes.
That said, the timing of your question is perfect since we are about to push a Tilt integration that allows you to create an isolation flow and use Tilt to hot-reload and update that flow in-place! :)
It's also such an interesting moment for you folks to show up on HN, I just shipped the first preview version of my Kubernetes operator(https://github.com/pier-oliviert/sequencer) that manages ephemeral environments. I can see some things that are similar with both our options as well as some things that are quite different.
Maybe if I had one question is: What made you go for Istio as the main network mesh?
Good luck with the launch!
Nothing too specific on choosing Istio, but it seemed like a popular and battle-tested option to start the implementation with. Kardinal is a fairly young project (about 2 months old), and we expect it to adopt more "service mesh backends" in the future.