The original vulnerability description is not worded very well, here's my understanding of what's going on:
1. Attacker crafts a malicious Chart.yaml containing arbitrary code
2. Replaces Chart.lock with a symlink pointing to a sensitive file (like .bashrc or other startup scripts)
3. When you run helm dependency update, Helm processes the malicious Chart.yaml and writes the payload to whatever file the symlink targets
4. Code executes when the targeted file is next used (e.g., opening a new shell)
Why This Works:
Helm follows the symlink during the dependency update process without validating the target, allowing arbitrary file writes outside the intended chart directory.
Can anyone explain in what setup an attacker who can create a symlink where Chart.lock was could not directly write .bashrc or similar? Is this related to how Git handles symlinks?
A symlink is just a special file that contains a string of text, it's not tightly bound to the target like a hard link. You can write anything into that string of text, including, say, "~/.bashrc". Then you can ship that symlink onto another system, and it suddenly points to your .bashrc.
Git just moves symlinks across systems as is, so yes, you can use git to deploy the exploit.
This has nothing to do with Git. A symlink can be packaged up in a tarball and shipped from one system to another. An attacker would need to create a malicious Chart.yaml file and a Chart.lock file pointing to another file. Then ship those to a system where dependencies are then updated.
This doesn't affect things like installing or upgrading a chart. Dependencies aren't updated at that time.
Helm is a program that allows users to creates packages which other users consume. Those packages contain files that are normally generated by Helm itself, but apparently if you alter your package definition by hand you can replace Chart.lock with a symlink.
As I'm typing this it's occurring to me that you probably shouldn't be able to do that. The fix they applied was to prevent the actual write from occurring when trying to write the lockfile and determining that the lockfile is a symlink. They could (should?) also validate that like, the package itself hasn't been screwed with in this manner.
Having read the CVE multiple times I am still unsure how 2. above happens? Is it possible through the malicious chart itself or is it a dependency for the CVE to be in play at all? And if the latter - what local process would write a symlink from a helm lock file to any kind of system start up script which doesn't point to a much bigger problem than this CVE?
The attacker creates a symlink (e.g., using `ln -s`) to another file. The attacker needs to create the malicious Chart.yaml file and symlink that the Chart.lock file points to.
Helm is an abomination, as the whole idea of using a text template engine to generate YAML is. And this vulnerability adds insult to injury ;)
Sorry, just can't really recover from trauma of counting spaces and messing up newlines, etc. when writing Helm templates. You know, Lisp "sucks" because "you need to count parenthesis" (you actually don't), yet Helm is a widely accepted technology where you need to count spaces for (n)indent ;)
I'm a dev that jumped to devops and one of my pet peeves will always be the lengths devops engineers go to avoid using a real programming language. Instead of interacting with all these APIs through python, ruby, lua, go, whatever they would rather build hodgepodge systems in bash, coreutils, curl (or wget. or both!) and jq (which is the worst). Or in the case of helm, just creating a half yaml/half Go SDK for generating YAML.
Even the helm infrastructure that I work in is completely wrapped in custom shell scripts that call all sorts of other commands to populate helm variables.
But yeah it's silly that helm templates require all sorts of {{ indent | 4 }} type incantations when the final YAML output is just sent through some kind of toJSON anyway.
You'll often find that if you write ops scripts in, say Python, it's largely calling external commands.
When that's the case, bash if often the better choice, especially if you know it well. It has an excellent REPL, is easy to trace and is already installed everywhere.
> the lengths devops engineers go to avoid using a real programming language
OK, but, you know... those tools were created by literal devs. Not in yaml, in a "real language"! So apparently devs thought they needed that.
The argument should be towards all people - we love creating new abstractions and "simplify things", but we suck at honest evaluation of the impact of our creations.
Still: I absolutely hate Helm templating and think that the very existence of "helpers" (even in a default chart!) is an abomination.
Yeah, vi has supported % as "jump between matching parenthesis" since it's original release in the 1970s, and vim by default will do simple parenthesis matching and highlighting, I don't see why everyone is so scared of touching lisp for these reasons with modern editors (if your editor doesn't support either of the above... maybe it's not modern enough?)
… and I think I'd argue that the parent's argument against the tooling would apply equally as well to those "other things", too.
The alternative here is something that manipulates the data structure directly. E.g., it might permit me to say:
my_config_map.data["key"] = some_string_value
(This is in some pseudo-imperative language, vs. the parent's Lisp, but that distinction isn't particular relevant to the core of their argument, I think.)
And then at the end, the thing itself takes care of converting the resulting objects to YAML, thus preventing me from inadvertently turning what is meant to be a string into something like an accidental YAML-injection that results in terrible errors because I miscounted the number of spaces to indent something.
The funny thing is helm is as good or bad as what you make it. When folks complain about helm they are actually complaining about their own self created charts or poor selection of charts they install.
I don’t think there is a lot of overlap between people who say lisp sucks because of the parens and people who are fine with using a template to generate yaml.
I like wrapping with kapp for this. However kustomize still skips hooks if you inflate a chart that uses them and there is value in discoverability of helm apps installed with clear versioning
You place malicious binaries outside the helm directory. Helm will now execute malicious code through the helm chart pointing outside the helm directory.
Don't I have already bigger problems if you have access to my file system to place there malicious code?
Is the danger here that one can get an execute permission? But if you can manipulate my helm chart why can you not also place the malicious code in the helm directory?
Yeah, there is a rather strong "downloading and executing arbitrary code from the Internet may lead to execution of arbitrary code" kind of vibe there.
I think that there are "legitimate" use cases for symlinks that read from outside the root, which at this point are probably looked upon even less favorably. It's likely that making the change you're proposing would be backwards incompatible.
I agree that it's not clearly explained why this isn't a concern though. A cursory search for other instances of os.WriteFile doesn't seem to surface any thorough controls...
edit: ok actually it looks like the lockfile is special because it's the only instance of helm itself directly writing a file on behalf of a package consumer
If you have a chart that has `deploy.yaml` symlinked to `/home/john/testcharts/redis/deploy.yaml`, that chart is clearly not going to work on anyone's machine except john's, so that chart is useless on anyone else's machine.
If you're saying the use-case is for charts that aren't distributed, well, I'm saying we should ban all symlinks on distribution (downloading and unpacking a chart should fail if it has symlinks outside of the root), and I just can't imagine any use-case where a distributed chart with external symlinks makes sense.
If this whole thing is about charts that aren't distributed, but local to some developer's machine, well, in that case who cares if the developer can pwn themselves by typing "ln -s ~/.bashrc Chart.lock", they could have just pwned themselves by typing "bash" even more quickly.
What is the attack scenario here? Where are the security boundaries? How does the attacker gets their repository with a symlink in it to the victim? Is Helm typically run as a privileged user? How would this work? And why doesn't the vulnerability description give answers to these questions?
Given the details in the article, I think even something as simple a templating a chart from a repository might be vuln., but it likely depends on a lot of exact specifics.
> Where are the security boundaries?
I expect templating does not result in LCE.
> How does the attacker gets their repository with a symlink in it to the victim?
The attacker owns the repository. They can serve whatever maliciousness in it they want. But should templating a malicious chart result in LCE?
> Is Helm typically run as a privileged user?
Enough so, yes, because the rendered result is often pushed to a k8s cluster. "Privileged" here might not be "root", but it might be "this user has k8s API access".
Imagine, e.g., that the attacker's LCE here might be "push ~/.kube to attacker".
> And why doesn't the vulnerability description give answers to these questions?
Familiarity with the tools involved is an normal assumption.
Pretty cool and nice find. I already have a "malicious" Chart.yaml in mind for this attack just based on the description of vuln.
Fortunately, my dotfiles are managed with nix so trying to write to those files on a read only partition will raise many red flags for me.
I don't use bash, but maybe should write a dummy .bashrc (and other start up script equivalents for fish) as some sort of canary.
If I happen to overlook the malicious shell script crafted in a dependency on helm chart, I would get nasty errors that a process was trying to write to a read only file.
Readit News
1. Attacker crafts a malicious Chart.yaml containing arbitrary code
2. Replaces Chart.lock with a symlink pointing to a sensitive file (like .bashrc or other startup scripts)
3. When you run helm dependency update, Helm processes the malicious Chart.yaml and writes the payload to whatever file the symlink targets
4. Code executes when the targeted file is next used (e.g., opening a new shell)
Why This Works: Helm follows the symlink during the dependency update process without validating the target, allowing arbitrary file writes outside the intended chart directory.
Git just moves symlinks across systems as is, so yes, you can use git to deploy the exploit.
This doesn't affect things like installing or upgrading a chart. Dependencies aren't updated at that time.
As I'm typing this it's occurring to me that you probably shouldn't be able to do that. The fix they applied was to prevent the actual write from occurring when trying to write the lockfile and determining that the lockfile is a symlink. They could (should?) also validate that like, the package itself hasn't been screwed with in this manner.
Sorry, just can't really recover from trauma of counting spaces and messing up newlines, etc. when writing Helm templates. You know, Lisp "sucks" because "you need to count parenthesis" (you actually don't), yet Helm is a widely accepted technology where you need to count spaces for (n)indent ;)
Even the helm infrastructure that I work in is completely wrapped in custom shell scripts that call all sorts of other commands to populate helm variables.
But yeah it's silly that helm templates require all sorts of {{ indent | 4 }} type incantations when the final YAML output is just sent through some kind of toJSON anyway.
When that's the case, bash if often the better choice, especially if you know it well. It has an excellent REPL, is easy to trace and is already installed everywhere.
OK, but, you know... those tools were created by literal devs. Not in yaml, in a "real language"! So apparently devs thought they needed that.
The argument should be towards all people - we love creating new abstractions and "simplify things", but we suck at honest evaluation of the impact of our creations.
Still: I absolutely hate Helm templating and think that the very existence of "helpers" (even in a default chart!) is an abomination.
The alternative here is something that manipulates the data structure directly. E.g., it might permit me to say:
(This is in some pseudo-imperative language, vs. the parent's Lisp, but that distinction isn't particular relevant to the core of their argument, I think.)And then at the end, the thing itself takes care of converting the resulting objects to YAML, thus preventing me from inadvertently turning what is meant to be a string into something like an accidental YAML-injection that results in terrible errors because I miscounted the number of spaces to indent something.
Though it's lacking in several ways, like good destroy functionality.
- You have access to my file system
- You have access to the helm repository
You place malicious binaries outside the helm directory. Helm will now execute malicious code through the helm chart pointing outside the helm directory.
Don't I have already bigger problems if you have access to my file system to place there malicious code?
Is the danger here that one can get an execute permission? But if you can manipulate my helm chart why can you not also place the malicious code in the helm directory?
No, helm is the one doing this part in the vuln. Chart.lock is made a symlink to some important file, and helm will happily write to it.
If you can manipulate my helm chart, why not just do the RCE directly in my kubernetes cluster or whatever?
Deleted Comment
Is the vulnerability that you ship a chart with `Chart.lock -> ../.bashrc`, and then helm writes to `Chart.lock`?
Why is the fix specific to Chart.lock (https://github.com/helm/helm/commit/76fdba4c8c2a4829a6b7abb4...), wouldn't the fix be instead that "A chart cannot contain any symlinks outside of its root"?
I agree that it's not clearly explained why this isn't a concern though. A cursory search for other instances of os.WriteFile doesn't seem to surface any thorough controls...
edit: ok actually it looks like the lockfile is special because it's the only instance of helm itself directly writing a file on behalf of a package consumer
If you have a chart that has `deploy.yaml` symlinked to `/home/john/testcharts/redis/deploy.yaml`, that chart is clearly not going to work on anyone's machine except john's, so that chart is useless on anyone else's machine.
If you're saying the use-case is for charts that aren't distributed, well, I'm saying we should ban all symlinks on distribution (downloading and unpacking a chart should fail if it has symlinks outside of the root), and I just can't imagine any use-case where a distributed chart with external symlinks makes sense.
If this whole thing is about charts that aren't distributed, but local to some developer's machine, well, in that case who cares if the developer can pwn themselves by typing "ln -s ~/.bashrc Chart.lock", they could have just pwned themselves by typing "bash" even more quickly.
Given the details in the article, I think even something as simple a templating a chart from a repository might be vuln., but it likely depends on a lot of exact specifics.
> Where are the security boundaries?
I expect templating does not result in LCE.
> How does the attacker gets their repository with a symlink in it to the victim?
The attacker owns the repository. They can serve whatever maliciousness in it they want. But should templating a malicious chart result in LCE?
> Is Helm typically run as a privileged user?
Enough so, yes, because the rendered result is often pushed to a k8s cluster. "Privileged" here might not be "root", but it might be "this user has k8s API access".
Imagine, e.g., that the attacker's LCE here might be "push ~/.kube to attacker".
> And why doesn't the vulnerability description give answers to these questions?
Familiarity with the tools involved is an normal assumption.
Dead Comment
Basic tech news?
Capitalist news?
Vulture Capitalist news?
Fortunately, my dotfiles are managed with nix so trying to write to those files on a read only partition will raise many red flags for me.
I don't use bash, but maybe should write a dummy .bashrc (and other start up script equivalents for fish) as some sort of canary.
If I happen to overlook the malicious shell script crafted in a dependency on helm chart, I would get nasty errors that a process was trying to write to a read only file.
Allowing LLMs to generate charts and what not via shell execution is a bad idea.
https://tytel.org/helm/
https://github.com/emacs-helm/helm
Although, the whole can of worms regarding synth/audio exploits is a pretty wild scene ..