If you can get malicious instructions into the context of even the most powerful reasoning LLMs in the world you'll still be able to trick them into outputting vulnerable code like this if you try hard enough.
I don't think the fact that small models are easier to trick is particularly interesting from a security perspective, because you need to assume that ANY model can be prompt injected by a suitably motivated attacker.
On that basis I agree with the article that we need to be using additional layers of protection that work against compromised models, such as robust sandboxed execution of generated code and maybe techniques like static analysis too (I'm less sold on those, I expect plenty of malicious vulnerabilities could sneak past them.)
The most "shocking" thing to me in the article is that people (apparently) think it's acceptable to run a system where content you've never seen can be fed into the LLM when it's generating code that you're putting in production. In my opinion, if you're doing that, your whole system is already compromised and you need to literally throw away what you're doing and start over.
Generally I hate these "defense in depth" strategies that start out with doing something totally brain-dead and insecure, and then trying to paper over it with sandboxes and policies. Maybe just don't do the idiotic thing in the first place?
When you say "content you've never seen," does this include the training data and fine-tune content?
You could imagine a sufficiently motivated attacker putting some very targeted stuff in their training material - think StuxNet - "if user is affiliated with $entity, switch goals to covert exfiltration of $valuable_info."
We started giving our (https://www.definite.app/) agent a sandbox (we use e2b.dev) and it's solved so many problems. It's created new problems, but net-net it's been a huge improvement.
Something like "where do we store temporary files the agent creates?" becomes obvious if you have a sandbox you can spin up and down in a couple seconds.
It wasn't, but the written version of it it is actually better than what I said in the room (since I got to think a little bit harder and add relevant links).
> The conventional wisdom that local, on-premise models offer a security advantage is flawed. While they provide data privacy, our research shows their weaker reasoning and alignment capabilities make them easier targets for sabotage.
Yeah, I'm not following here. If you just run something like deepseek locally, you're going to be okay provided you don't feed it a bogus prompt.
Outside of a user copy-pasting a prompt from the wild, or break isolation by giving it access to outside resources, the conventional wisdom holds up just fine. The operator and consumption of 3rd party stuff are weak-points for all IT, and have been for ages. Just continue to train folks to not do insecure things, and re-think letting agents go online for anything/everything (which is arguably not a local solution anyway).
Freeform plaintext (not an executable/script) being an attack vector is new, outside of parser vulns. Providing context through tickets, docs, etc is now a non-obvious security liability.
It is still an important attack vector to be aware of regardless of how unrealistic you believe it to be. Many powerful hacks come from very simple and benign appearing starting points.
All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed.
If you have absolutely no idea what you're doing, well, then it doesn't really matter in the end, does it? You're never gonna recognize any security vulnerabilities (as has happened many times with LLM-assisted "no-code" platforms and without any actual malicious intent), and you're going to deploy unsafe code either way.
Sure, you can simplify these observations into just codegen. But the real observation is not that these models are more susceptible to fail when generating code, but that they are more susceptible to jailbreak-type attacks that most people have come to expect to be handled by post training.
Having access to open models is great, and even if their capabilities are somewhat lower than the closed-source SoTA models, and we should be aware of the differences in behavior.
> more susceptible to jailbreak-type attacks that most people have come to expect to be handled by post training
the keyword here is "more". The big models might not be quite as susceptible to them, but they are still susceptible. If you expect these attacks to be fully handled, then maybe you should change your expectations.
> All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed.
Well this is wrong. And it's exactly this type of thinking why people will get absolutely burned by this.
First off the fact they chose obvious exploits for explanatory purposes doesn't mean this attack only supports obvious exploits...
And to your second point of "review the code before you deploy to prod", the second attack did not involve deploying any code to prod. It involved an LLM reading a reddit comment or github comment and immediately executing.
People not taking security seriously and waving it off as trivial is what's gonna make this such a terrible problem.
I thought that local LLMs means they run on local computers, without being exposed to the internet.
If an attacker can exploit a local LLM, means it already compromised you system and there are better things they can do than trick the LLM to get what they can get directly.
LLMs don't have any distinction between instructions & data. There's no "NX" bit. So if you use a local LLM to process attacker-controlled data, it can contain malicious instructions. This is what Simon Willson's "prompt injection" means: attackers can inject a prompt via the data input. If the LLM can run commands (i.e. if it's an "agent") then prompt injection implies command execution.
>LLMs don't have any distinction between instructions & data
And this is why prompt injection really isn't a solvable problem on the LLM side. You can't do the equivalent of (grep -i "DROP TABLE" form_input). What you can do is not just blindly execute LLM generated code.
I guess if you were using the LLM to process data from your customers, e.g. categorise their emails, then this argument would hold that they might be more risky.
Local LLMs may not be exposed to the internet, but if you want them to do something useful you're likely going to hook them up to an internet-accessing harness such as OpenCode or Claude Code or Codex CLI.
Fair enough. Forgive my probably ignorance, but if Claude Code can be attacked like this, doesn’t that means that also foundation LLMs are vulnerable to this, and is not a local LLM thing?
yes and I think better local sandboxing can help out in this case, it’s something ive been thinking about a lot and more and more seems to be the right way to run these things
Welcome to corporate security. "If an attacker infiltrates our VPN and gets on the network with admin credentials and logs into a workstation..." Ya, no shit, thanks Mr Security manager, I will dispose of all of our laptops.
Yeah, I don't understand what the hosting environment of the LLM has to do with this. Seems like FUD from people with an interest in SaaS LLMs.
If you're leveraging an LLM that can receive arbitrary inputs from vetted sources, and allowing that same LLM to initiate actions that target your production environment, you are exposing yourself to the same risk regardless of whether the LLM itself is running on your servers or someone else's.
The underlying problem here is giving any model direct access to your primary system. The model should be working in a VM or container with limited privileges.
This is like saying it's safer to be exposed to dangerous carcinogenic fumes than nerve gas, when the solution is wearing a respirator.
Also what are you doing allowing someone else to prompt your local LLM?
This vulnerability comes from allowing the AI to read untrusted data (usually documentation) from the Internet. For LLMs the boundary between "code" and "data" isn't as clear as it used to be since they will follow instructions written in human language.
The "lethal trifecta" sounds catchy but I don't believe it accurately characterizes the risks of LLMs.
In theory any two of the trifecta is fine, but practically speaking I think you only need "ability to communicate with the outside," or maybe not even that. Business logic is not really private data anymore. Most devs are likely one `npm update` away from their LLM getting a new command from some transitive dependency.
The LLM itself is also a giant blackbox of unverifiable untrusted data, so I guess you just have to cross your fingers on that one. Maybe your small startup doesn't need to be worried about models being seeded with adversarial training data, but if I were say Coinbase I'd think twice before allowing LLM access to anything.
Readit News
I don't think the fact that small models are easier to trick is particularly interesting from a security perspective, because you need to assume that ANY model can be prompt injected by a suitably motivated attacker.
On that basis I agree with the article that we need to be using additional layers of protection that work against compromised models, such as robust sandboxed execution of generated code and maybe techniques like static analysis too (I'm less sold on those, I expect plenty of malicious vulnerabilities could sneak past them.)
Coincidentally I gave a talk about sandboxing coding agents last night: https://simonwillison.net/2025/Oct/22/living-dangerously-wit...
Generally I hate these "defense in depth" strategies that start out with doing something totally brain-dead and insecure, and then trying to paper over it with sandboxes and policies. Maybe just don't do the idiotic thing in the first place?
You could imagine a sufficiently motivated attacker putting some very targeted stuff in their training material - think StuxNet - "if user is affiliated with $entity, switch goals to covert exfiltration of $valuable_info."
Something like "where do we store temporary files the agent creates?" becomes obvious if you have a sandbox you can spin up and down in a couple seconds.
Yeah, I'm not following here. If you just run something like deepseek locally, you're going to be okay provided you don't feed it a bogus prompt.
Outside of a user copy-pasting a prompt from the wild, or break isolation by giving it access to outside resources, the conventional wisdom holds up just fine. The operator and consumption of 3rd party stuff are weak-points for all IT, and have been for ages. Just continue to train folks to not do insecure things, and re-think letting agents go online for anything/everything (which is arguably not a local solution anyway).
If you have absolutely no idea what you're doing, well, then it doesn't really matter in the end, does it? You're never gonna recognize any security vulnerabilities (as has happened many times with LLM-assisted "no-code" platforms and without any actual malicious intent), and you're going to deploy unsafe code either way.
Having access to open models is great, and even if their capabilities are somewhat lower than the closed-source SoTA models, and we should be aware of the differences in behavior.
the keyword here is "more". The big models might not be quite as susceptible to them, but they are still susceptible. If you expect these attacks to be fully handled, then maybe you should change your expectations.
Deleted Comment
Well this is wrong. And it's exactly this type of thinking why people will get absolutely burned by this.
First off the fact they chose obvious exploits for explanatory purposes doesn't mean this attack only supports obvious exploits...
And to your second point of "review the code before you deploy to prod", the second attack did not involve deploying any code to prod. It involved an LLM reading a reddit comment or github comment and immediately executing.
People not taking security seriously and waving it off as trivial is what's gonna make this such a terrible problem.
right, so you shouldn't give the LLM access to execute arbitrary commands without review.
I thought that local LLMs means they run on local computers, without being exposed to the internet.
If an attacker can exploit a local LLM, means it already compromised you system and there are better things they can do than trick the LLM to get what they can get directly.
And this is why prompt injection really isn't a solvable problem on the LLM side. You can't do the equivalent of (grep -i "DROP TABLE" form_input). What you can do is not just blindly execute LLM generated code.
I will fight and die on the hill that "LLMs don't need the internet to be useful"
If you're leveraging an LLM that can receive arbitrary inputs from vetted sources, and allowing that same LLM to initiate actions that target your production environment, you are exposing yourself to the same risk regardless of whether the LLM itself is running on your servers or someone else's.
This is like saying it's safer to be exposed to dangerous carcinogenic fumes than nerve gas, when the solution is wearing a respirator.
Also what are you doing allowing someone else to prompt your local LLM?
If you are executing local malicious/unknown code for reasons you need to read this...
Sounds like the Open Source model did exactly as it was prompted, where the "Closed" AI did the wrong thing and disregarded the prompt.
That means the closed model was actually the one that failed the alignment test.
In theory any two of the trifecta is fine, but practically speaking I think you only need "ability to communicate with the outside," or maybe not even that. Business logic is not really private data anymore. Most devs are likely one `npm update` away from their LLM getting a new command from some transitive dependency.
The LLM itself is also a giant blackbox of unverifiable untrusted data, so I guess you just have to cross your fingers on that one. Maybe your small startup doesn't need to be worried about models being seeded with adversarial training data, but if I were say Coinbase I'd think twice before allowing LLM access to anything.