Projects like this reiterate just how important it is, from a security perspective, to ensure your production services are running in containers without an included shell. If an attacker can get a shell, they can do pretty much anything.
Debug containers are now a stable feature in Kubernetes. It honestly boggles my mind how companies will throw so much time, money, and effort into the cybersecurity product du jour when they can get the vast majority of the value by moving everything into distroless, shell-less containers running on managed VMs that are optimized for container workloads.
Debug containers have been a bit of a let down in UX. It's rather difficult to do sort of basic stuff, and a lot of stuff is hidden behind flags.
E.g., if you just sort of roll with the defaults, you're dropped into a pod in a very confused state: `ps -ef` says nothing in your debugee is running, and the filesystem of the debugee is nowhere to be found.
You can work around both of those (the first is --target, but the latter requires an intricate SO answer¹) but its the sort of thing that would be nicer out of the box?
The node debugging mode is a bit better: by default, puts you in at least the host pidns, and mounts the host FS.
I would agree, but so much of day-to-day Kubernetes is arcane CLI commands to begin with. Other stuff that is non-trivial to do on the CLI but comes up in most reasonable production deployments:
* Rotating secrets without exposing the secret to the shell history file (hint: kubectl apply -f can take - to signify atdin, but not kubectl patch!)
* Ensuring your edits to a ConfigMap pass application-level validation (i.e. your configuration changes won't crash your app, not just that it's a valid ConfigMap)
* Anything to do with user auth or RBAC
* Scaling the default persistent volume size of a StatefulSet
The truth is that Kubernetes is a platform, and just like how most people don't want to run a bare copy of Bash or VIM on their laptop, people will figure out aliases, one-liners, and other functions to help make them effective. So some of working effectively with Kubernetes means, yes, building your own custom debug containers, and writing your own helper shell stuff.
I've also been disappointed with debug containers. They are often not useful for debugging trick production-only issues because so many of those issues are related to container state, which can be (often is) different inside the container. Certain languages/platforms and developer discipline are better about this than others, like if you're using functional/immutable languages then it's less of an issue.
For applications that aren't super high security, I've been really appreciating using immutable hosts (that get regularly updated/rotated), along with CI/CD that is constantly rebuilding from source, applying latest software updates, and deploying the latest version of the app. Combined with other tools like scanners, and de-bloating your images, it really raises the height of the fruit.
I work on multi-tenant k8s clusters at CDNs and used to work at Rancher and have seen just about every multi-tenant / federated deployment there is, nasa, meta, etc. Stuff where even the hardware vfio paths mounting the nic or gpu channels keep users apart from one another, and the entire path out of the cluster are completely apart from k8s and any userspace and would be something like multus as a shim- what exactly are you referring to that can cgroup hop via a shell that we're not currently mitigating? It's the hardware being infected by something we worry about at this level.
A lot of the CDNs even use tools like kubevirt where the segmentation is even further. And then we have gvisor, firecracker, etc.
I admittedly haven't touched k8s code since 1.18 but I can't think of anything like you're referring to and I definitely would like to know about it.
Hey, great work! Reading through I started wonder how necessary the loadables are? It'd be fun to have one that's not dependent on loadbales, even if it's not as clean. E.g. could mktemp be replaced with a timestamp named directory or something? Can rm be avoided by just allowing garbage to pile up? Is finfo something that can be worked around in some way?
I suspect lots of people have written a "use tcpserver or inetd and feed stdout to a shell script" antics.
The thing is, shell can't cope with nulls -- if you do something like
n=$(gzip -9 < /etc/passwd)
gzip -9 < /etc/passwd | sum
echo "$n" | sum
This falls apart because shell just can't deal with nulls.
You can probably hack around all those issues, and may not run into this too much, at first, in a web server, but golly you'll pretty quickly fall into a pit.
Will sanitize strings of non-printable characters. While it is true that you can't have nulls inside bash variables, your example actually contains the correct syntax if you just remove the first and last lines.
well, sure, you can use some external program do process the stdout; then it's no longer "pure bash" which is fine, nobody grades ingots of script based on if they're 90% or 99% or 70% "pure" shell.
But -- importantly -- running
n=$(gzip -9 < /etc/passwd | tr -dc '[[:print:]]')
may process the nulls, but is it reversible? Can I now send $n into gzip -d and get whatever I put into it out?
I can do things that are reversible --
n=$(gzip -9 < /etc/passwd | base64 )
But now I can't process the output "natively" except by calling base64 every time.
And maybe I've gotten myself into this hole because sometimes the contents of $n have nulls and other times not?
Pure shell is a road to madness. Don't ask me how I know...
The conectiva Linux distro had a programmer that wrote a book on Shell Script in which he implemented a bash server, but as apache cgi scripts.
I learned to properly program with that book
A long time ago I made a similarly pure bash version of something like tcpdump just parsing various packets and protocols off a raw socket. I wish I still had that code somewhere. It was pretty much the slowest and least-robust thing of all time but was kind of fun to play around with.
Readit News
Debug containers are now a stable feature in Kubernetes. It honestly boggles my mind how companies will throw so much time, money, and effort into the cybersecurity product du jour when they can get the vast majority of the value by moving everything into distroless, shell-less containers running on managed VMs that are optimized for container workloads.
E.g., if you just sort of roll with the defaults, you're dropped into a pod in a very confused state: `ps -ef` says nothing in your debugee is running, and the filesystem of the debugee is nowhere to be found.
You can work around both of those (the first is --target, but the latter requires an intricate SO answer¹) but its the sort of thing that would be nicer out of the box?
The node debugging mode is a bit better: by default, puts you in at least the host pidns, and mounts the host FS.
¹https://stackoverflow.com/questions/73355970/how-to-get-acce...
I would agree, but so much of day-to-day Kubernetes is arcane CLI commands to begin with. Other stuff that is non-trivial to do on the CLI but comes up in most reasonable production deployments:
The truth is that Kubernetes is a platform, and just like how most people don't want to run a bare copy of Bash or VIM on their laptop, people will figure out aliases, one-liners, and other functions to help make them effective. So some of working effectively with Kubernetes means, yes, building your own custom debug containers, and writing your own helper shell stuff.For applications that aren't super high security, I've been really appreciating using immutable hosts (that get regularly updated/rotated), along with CI/CD that is constantly rebuilding from source, applying latest software updates, and deploying the latest version of the app. Combined with other tools like scanners, and de-bloating your images, it really raises the height of the fruit.
Seems only marginally better than using nsenter on a privileged container to just go muck with the host
A lot of the CDNs even use tools like kubevirt where the segmentation is even further. And then we have gvisor, firecracker, etc.
I admittedly haven't touched k8s code since 1.18 but I can't think of anything like you're referring to and I definitely would like to know about it.
Thanks.
I'm really happy to see somebody shared it on hackernews :D If you have some questions, feel free to ask me
You could avoid loadables.
Finfo <- load file inside a variable and get the size Mktemp <- like you said with timestamp Rm <- with a fifo or variable
The thing is, shell can't cope with nulls -- if you do something like
This falls apart because shell just can't deal with nulls.You can probably hack around all those issues, and may not run into this too much, at first, in a web server, but golly you'll pretty quickly fall into a pit.
But -- importantly -- running
may process the nulls, but is it reversible? Can I now send $n into gzip -d and get whatever I put into it out?I can do things that are reversible --
But now I can't process the output "natively" except by calling base64 every time.And maybe I've gotten myself into this hole because sometimes the contents of $n have nulls and other times not?
Pure shell is a road to madness. Don't ask me how I know...
Show HN: A pure bash web server. No netcat, socat, etc. - https://news.ycombinator.com/item?id=29794979 - Jan 2022 (97 comments)
https://www.amazon.com.br/Script-Profissional-Aurelio-Marinh...
cool project