I'm lazy and I don't like having to remember "the right way" to run something, so my solution is directories and wrappers. I keep a directory for every environment (dev, stage, prod, etc) for every account I manage.

  env/
      account-a/
                dev/
                stage/
                prod/
      account-b/
                dev/
                stage/
                prod/

I keep config files in each directory. I call a wrapper script, cicd.sh, to run certain commands for me. When I want to deploy to stage in account-b, I just do:

  ~ $ cd env/account-b/stage/
  ~/env/account-b/stage $ cicd.sh deploy
  cicd.sh: Deploying to account-b/stage ...

The script runs ../../../modules/deploy/main.sh and passes in configs from the current directory ("stage") and the previous directory ("account-b"). Those configs are hard-coded with all the correct variables. It's impossible for me to deploy the wrong thing to the wrong place, as long as I'm in the right directory.

I use this model to manage everything (infrastructure, services, builds, etc). This has saved my bacon a couple times; I might have my AWS credentials set up for one account (export AWS_PROFILE=prod) but trying to deploy nonprod, and the deploy immediately fails because the configs had hard-coded values that didn't match my environment.

Agree, this is a huge pain point when dealing with multiple clusters. I wrote a wrapper for `kubectl` that displays the current context for `apply` & `delete` and prompts me to confirm the command. It's not perfect, but it's saved me a lot of trouble already — but encouraging other members of the team to have a similar setup is another story.

Here's the script (along with a bunch of extra utils): https://github.com/pch/dotfiles/blob/master/kubernetes/utils...

A lot of people like kubectx. Or specifying contexts. Personally I hate both approaches.

For the several dozen clusters that I manage, I have separate kubeconfig files for each and I use the --kubeconfig flag.

It's explicit and I have visual feedback in the command I run for the cluster I'm running against, by short name. No stupidly long contexts.

Not a perfect solution but I add a prompt signaling both my current namespace and cluster, along with some safeguards for any changes on our production environment. In practice I haven't deployed something wrongfully in production ever.

I use a custom written script but I've used this one in the past - its pretty nice.

https://github.com/jonmosco/kube-ps1/blob/master/kube-ps1.sh

We partially resolve this by having different namespaces in each of our environments. Nothing is ever run in the 'default' namespace.

So if we think we're targeting the dev cluster and run 'kubectl -n dev-namespace delete deployment service-deployment' but our current context is actually pointing to prod then we trigger an error as there is no 'dev-namespace' in prod.

Obviously we can associate specific namespaces to contexts to traverse this safety net but it can help in some situations.

direnv is our magic sauce for this. We enforce that all devs store the current context in an environment variable (KUBECTL_CONTEXT), and define the appropriate kubectl alias to always use that variable as the current context. To do stuff in a cluster, cd into that cluster’s directory, and direnv will automatically set the correct context. I also change prompt colors based on the current context.

(This way, the worst you can do is re-apply some yaml that should’ve already been applied in that cluster anyway)

We also have a Makefile in every directory, where the default pseudo-target is the thing you want 99% of the time anyway: kustomize build | kubectl apply -f -

I wrote a convoluted tool for this problem which isolates kubectl environments in docker containers: https://github.com/forestgagnon/kparanoid.

This approach allows the convenience of short, context-free commands without compromising safety, because the context info in the shell prompt can be relied on, due to the isolation.

There are some things which don't work well inside a docker container (port-forwarding for example), but it does make it simple to have isolated shell history, specific kubectl versions, etc.

When I was running the internal k8s clusters at a previous workplace, I simply got into the habit of compulsively running `kubectl config current-context` to check which one of the 50+ clusters I was currently connected to (designated test clusters for *playing with cluster infra", designated clusters for "devs playing around", designated prod clusters, with segregation between "batch-like" and "interactive" workloads, as we needed to treat the nodes differently in those, designated "run the CI/CD pipelines" clusters, as they needed different RBAC, ... and then duplicate between multiple data centres).

thanks for starting that thread, context is a major hurdle for beginners.

I myself am quite happy with the basics, but have an alias on k=kubectl and set-context that without argument displays the current-context. Before doing anything I rename or edit contexts in .kube/config to have a minimal amount of characters to type for the target ("proj-prod"). Using -l name= is another help in filtering, jsonpath and jq too.. as years ago with using the cli prompts with database products, building up muscle memory also gave me opportunity to grok the concepts at the same time.

After some attempts with different tooling, I came to like kubernetes for what it can do.

k9s is a fantastic tool. It's a CLI GUI written in go.

I like the spirit of this but for dealing with multiple clusters, kubectx is pretty standard, always returns highlighting where you are and we don't have to type in the cluster name in every command. Also avoiding "kubectl delete" seems such a narrow case, I can still delete with "k scale --replicas=0" and possibly many other ways; at this point you are better of with a real RBAC implementation.

Every member of our ops team has the following PS1 var in .bashrc to prevent such accidents:

  PS1='\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]\[\033[01;33m\] [`kubectl config current-context| rev | cut -d_ -f1 | rev`] \[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\] $ '

couldn't that be solved by not allowing production access to those clusters? most k8s providers should allow role based access (read/write/deploy)

Nice trick, I usually use sleep 10000000 as the command.

Not only containers, it can also forward services!

omg TIL `k explain foo` and `k explain foo.spec` is a thing. thank you for this!!!

Wait What

  <type>.<fieldName>[.<fieldName>]
  
 Add the --recursive flag to display all of the fields at once without descriptions. Information about each field is

  # Get the documentation of a specific field of a resource
  kubectl explain pods.spec.containers

I could be wrong here, but I think `helm lint` just checks that the chart is formed correctly — Go templating and all.

I don't think it validates the Kubernetes resources.

Here's an example:

$ helm create foo

$ cd foo

Then change "apiVersion" in deployment.yaml to "apiVersion: nonsense"

In the linting, I got

$ helm lint ==> Linting . [INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

$ helm template . | kubeval -

ERR - foo/templates/deployment.yaml: Failed initializing schema https://kubernetesjsonschema.dev/master-standalone/deploymen...: Could not read schema from HTTP, response status is 404 Not Found

Along those lines, this was an interesting statement:

"You should learn how to use these commands, but they shouldn't be a regular part of your prod workflows. That will lead to a flaky system."

It seems like there's some theory vs. practice tension here. In theory, you shouldn't need to use these commands often, but in practice, you should be able to do them quickly.

How often is it the case in reality that a team of Kubernetes superheroes, well versed in these commands, is necessary to make Continuous Integration and/or Continuous Deployment work?