I guess I don't really get the attack. The idea seems to be that if you give your Claude an access token, despite what you tell it that it's for, Claude can be convinced to use it for anything that it's authorized for.
I think that's probably something anybody using these tools should always think. When you give a credential to an LLM, consider that it can do up to whatever that credential is allowed to do, especially if you auto-allow the LLM to make tool use calls!
But GitHub has fine-grained access tokens, so you can generate one scoped to just the repo that you're working with, and which can only access the resources it needs to. So if you use a credential like that, then the LLM can only be tricked so far. This attack wouldn't work in that case. The attack relies on the LLM having global access to your GitHub account, which is a dangerous credential to generate anyway, let alone give to Claude!
The issue here (which is almost always the case with prompt injection attacks) is that an LLM has access to attacker-controlled data, sensitive information, and a data exfiltration capability.
THe "cardinal rule of agent design" should be that an LLM can have access to at most two of these during one session. To avoid security issues, agents should be designed in a way that ensures this.
For example, any agent that accesses an issue created by an untrusted party should be considered "poisoned" by attacker-controlled data. If it then accesses any private information, its internet access capability should be severely restricted or disabled altogether until context is cleared.
In this model, you don't need per-repo tokens. As long as the "cardinal rule" is followed, no security issue is possible.
Sadly, it seems like MCP doesn't provide the tools needed to ensure this.
Genuine question - can we even make a convincing argument for security over convenience to two generations of programmers who grew up on corporate breach after corporate breach with just about zero tangible economic or legal consequences to the parties at fault? Presidential pardons for about a million a pop [1]?
What’s the cassus belli to this younger crop of executives that will be leading the next generation of AI startups?
> The "cardinal rule of agent design" should be that an LLM can have access to at most two of these during one session. To avoid security issues, agents should be designed in a way that ensures this.
Then don't give it your API keys? Surely there's better ways to solve this (like an MCP API gateway)?
> an LLM has access to attacker-controlled data, sensitive information, and a data exfiltration capability
> THe "cardinal rule of agent design" should be that an LLM can have access to at most two of these during one session
I still don't really get it. Surely the older, simpler, and better cardinal rule is that you just don't expose any service to the Internet that you have given access to your private data, unless you directly control that service and have a very good understanding of its behavior.
I might reword:
"attacker-controlled data, sensitive information, and a data exfiltration capability"
to:
"attacker-controlled data and privileged operations (e.g. sensitive information acces+data exfiltration or ability to do operations to production system)"
... is probably a bit unfair. From what I've seen the protocol is generally neutral on the topic of security.
But the rush to AI does tend to stomp on security concerns. Can't spend a month tuning security on this MCP implementation when my competition is out now, now, now! Go go go go go! Get it out get it out get it out!
That is certainly incompatible with security.
The reason anyone cares about security though is that in general lacking it can be more expensive than taking the time and expense to secure things. There's nothing whatsoever special about MCPs in this sense. Someone's going to roll snake eyes and discover that the hard way.
Can you give me more resources to read about this? It seems like it would be very difficult to incorporate web search or anything like that in Cursor or another IDE safely.
I don't know that this is a sustainable approach. As LLMs become more capable and are able to do the functions that a real human employee is doing they will need similar access that a normal human employee would have. Clearly not all employees have access to everything, but there is clearly a need for some broader access. Maybe we should be considering human type controls. If you are going to give broader access then you need X, Y and Z to do it like it requests temporary access from a 'boss' LLM etc etc. There are clear issues with this approach but humans also have these issues too (social engineering attacks work all too well). Is there potentially a different pattern that we should be exploring now?
I agree, one of the issues are tokens with too broad permission sets. However, at the same time, people want general agents which do not have to be unlocked on a repository-by-repository basis. That's why they give them tokens with those access permissions, trusting the LLM blindly.
Your caution is wise, however, in my experience, large parts of the eco-system do not follow such practices. The report is an educational resource, raising awareness that indeed, LLMs can be hijacked to do anything if they have the tokens, and access to untrusted data.
The solution: To dynamically restrict what your agent can and cannot do with that token. That's precisely the approach we've been working on for a while now [1].
If you look at Github's fine-grained token permissions then I can totally imagine someone looking at the 20-30 separate scopes and thinking "fuck this" while they back out and make a non-expiring classic token with access to everything.
It's one of those things where a token creation wizard would come in really handy.
This is applicable to those deployment services like Railway which require access to all the GitHub repositories even though we need to deploy only a single project. In that regard Netlify respects access to just the repository we want to deploy. GitHub shouldn't approve the apps which don't respect the access controls.
We had a bug bounty program manager who didn’t screen reports before sending them to each team as urgent tickets.
80% of the tickets were exactly like you said: “If the attacker could get X, then they can also do Y” where “getting X” was often equivalent to getting root on the system. Getting root was left as an exercise to the reader.
Were talking about githubs token system here... by the time you have generated the 10th one of these and its expiring or you lost them along the way and re-generated them your just smashing all the buttons to get through it as fast and as thoughtlessly as possible.
If you make people change their passwords often, and give them stupid requirements they write it down on a post it and stick it on their monitor. When you make your permissions system, or any system onerous the quality of the input declines to the minimal of effort/engagement.
Usability bugs are still bugs... it's part of the full stack that product, designers and developers are responsible for.
Sounds like confused deputy and is what capability-based systems solve. X should not be allowed to do Y, but only what the user was allowed to do in the first place (X is only as capable as the user, not more.)
Yea - I honestly don't get why a random commenter on your GitHub Repo should be able to run arbitrary prompts on a LLM which the whole "attack" seems to be based on?
Yes, if you let the chatbot face users you have to assume that the chatbot will be used for anything it is allowed to do. It's a convenience layer op top of your api. It's not an api itself. Clearly?
Well you're not giving the access token to Claude directly. The token is private to the MCP server and Claude uses the server's API, so the server could (should) take measures to prevent things like this from happening. It could notify the user whenever the model tries to write to a public repo and ask for confirmation, for instance.
Probably the only bulletproof measure is to have a completely separate model for each private repo that can only write to its designated private repo, but there are a lot of layers of security one could apply with various tradeoffs
Another type of attack waiting to happen is a malicious prompt in a url where an attacker could make the model do a curl request to post sensitive information
They exploit the fact the llm will do anything it can to anyone.
These tools cant exist securely as long as the llm doesn't reach at least the level of intelligence of a bug that can make decisions about access control and knows the concept of lying and bad intent
Even human level intelligence (whatever that means) is not enough. Social engineering works fine on our meat brains, it will most probably work on llms for foreseeable non-weird non-2027-takeoff-timeline future.
Based on “bug level of intelligence”, I (perhaps wrongly) infer that you don’t believe in possibility of a takeoff. In case it is even semi-accurate, I think llms can be secure, but, perhaps, humanity will be able to interact with such secure system for not so long time
Isn't the problem that the LLM can't differentiate between data and instructions? Or, at least in its current state? If we just limit it's instructions to what we / the MCP server provides, but don't let it eval() additional data it finds along the way, we wouldn't have this exploit – right?
Yes they can. If the token you give the LLM isn't permitted to access private repos you can lie all you want, it still can't access private repos.
Of course you shouldn't give an app/action/whatever a token with too lax permissions. Especially not a user facing one. That's not in any way unique to tools based on LLMs.
I've noticed this as a trend with new technology. People seem to forget the most basic things as if they don't apply because the context is new and special and different. Nope, you don't magically get to ignore basic security practices just because you're using some new shiny piece of tech.
See also: the cryptocurrency space rediscovering financial fraud and scams from centuries ago because they didn't think their new shiny tech needed to take any lessons from what came before them.
One of the authors here. Thanks for posting. If you are interested in learning more about MCP and agent security, check out some of the following resources, that we have created since we started working on this:
I think from security reasoning perspective: if your LLM sees text from an untrusted source, I think you should assume that untrusted source can steer the LLM to generate any text it wants. If that generated text can result in tool calls, well now that untrusted source can use said tools too.
I find it unsettling from a security perspective that securing these things is so difficult that companies pop up just to offer guardrail products. I feel that if AI companies themselves had security conscious designs in the first place, there would be less need for this stuff. Assuming that product for example is not nonsense in itself already.
I wonder if certain text could be marked as unsanitized/tainted and LLMs could be trained to ignore instructions in such text blocks, assuming that's not the case already.
This somewhat happens already, with system messages vs assistant vs user.
Ultimately though, it doesn't and can't work securely. Fundamentally, there are so many latent space options, it is possible to push it into a strange area on the edge of anything, and provoke anything into happening.
Think of the input vector of all tokens as a point in a vast multi dimensional space. Very little of this space had training data, slightly more of the space has plausible token streams that could be fed to the LLM in real usage. Then there are vast vast other amounts of the space, close in some dimensions and far in others at will of the attacker, with fundamentally unpredictable behaviour.
After I wrote the comment, I pondered that too (trying to think examples of what I called "security conscious design" that would be in the LLM itself). Right now and in near future, I think I would be highly skeptical even if an LLM was marketed as having such feature of being able to see "unsanitized" text and not be compromised, but I could see myself not 100% dismissing such thing.
If e.g. someone could train an LLM with a feature like that and also had some form of compelling evidence it is very resource consuming and difficult for such unsanitized text to get the LLM off-rails, that might be acceptable. I have no idea what kind of evidence would work though. Or how you would train one or how the "feature" would actually work mechanically.
Trying to use another LLM to monitor first LLM is another thought but I think the monitored LLM becomes an untrusted source if it sees untrusted source, so now the monitoring LLM cannot be trusted either. Seems that currently you just cannot trust LLMs if they are exposed at all to unsanitized text and then can autonomously do actions based on it. Your security has to depend on some non-LLM guardrails.
I'm wondering also as time goes on, agents mature and systems start saving text the LLMs have seen, if it's possible to design "dormant" attacks, some text in LLM context that no human ever reviews, that is designed to activate only at a certain time or in specific conditions, and so it won't trigger automatic checks. Basically thinking if the GitHub MCP here is the basic baby version of an LLM attack, what would the 100-million dollar targeted attack look like. Attacks only get better and all that.
No idea. The whole security thinking around AI agents seems immature at this point, heh.
> LLMs could be trained to ignore instructions in such text blocks
Okay, but that means you'll need some way of classifying entirely arbitrary natural-language text, without any context, whether it's an "instruction" or "not an instruction", and it has to be 100% accurate under all circumstances.
Maybe, but I think the application here was that Claude would generate responsive PRs for github issues while you sleep, which kind of inherently means taking instructions from untrusted data.
A better solution here may have been to add a private review step before the PRs are published.
You use prompt and mark correctly the input as <github_pr_comment> and clearly state read and never consider as prompt.
But the attack is quite convoluted. Do you still remember when we talked prompt injection in chat bots. It was a thing 2 years ago! Now MCP is buzzing...
> I feel that if AI companies themselves had security conscious designs in the first place, there would be less need for this stuff.
They do, but this "exploit" specifically requires disabling them (which comes with a big fat warning):
> Claude then uses the GitHub MCP integration to follow the instructions. Throughout this process, Claude Desktop by default requires the user to confirm individual tool calls. However, many users already opt for an “Always Allow” confirmation policy when using agents, and stop monitoring individual actions.
It's been such a long standing tradition in software exploits that it's kind of fun and facepalmy when it crops up again in some new technology. The pattern of "take user text input, have it be tainted to be interpreted as instructions of some kind, and then execute those in a context not prepared for it" just keeps happening.
SQL injection, cross-site scripting, PHP include injection (my favorite), a bunch of others I'm missing, and now this.
How is this considered an "exploit"? You give the agent a token that allows it to access a private repository. MCPs are just API servers. If you don't want something exposed in that API, don't grant them permissions to do so.
As many do, I also jumped to the comment section before actually reading the article.
If you do the same, you will quickly notice that this article features an attack. A malicious issue is posted on GitHub, and the issue features a LLM prompt that is crafted to leak data. When the owner of the GitHub account triggers the agent, the agent acts upon the malicious prompt on behalf of the repo owner.
I read it, and "attack" does not make sense. If you grant access to MCP to access some data (data you want anybody has access to like public repos, and data that only you want to access to like private repos), you will always be able to craft the prompt that will "leak" the data you are only supposed to access. That's not surprising at all. The only way to prevent these kind of "leaks" is not to provide the data feed with private data to the agent.
So it’s the e-mail exploit? If you e-mail someone and tell them to send you their password and they do, you suddenly have their password!? This is a very serious exploit in e-mail and need to be patched so it becomes impossible to do.
This "exploits" human fallibility, hence it is an exploit. The fallibility being users blindly buying into the hype and granting full access to their private Github repos thinking it is safe.
I'm going to be rather pedantic here given the seriousness of the topic. It's important that everyone understand how risky running a tool executing AI is, exactly.
Agents run various tools based on their current attention. That attention can be affected by the tool results from the tools they ran. I've even noted they alter the way they run tools by giving them a "personality" up front. However, you seem to argue otherwise, that it is the user's fault for giving it the ability to access the information to begin with, not the way it reads information as it is running.
This makes me think of several manipulative tactics to argue for something that is an irrational thought:
Stubborn argumentation despite clear explanations: Multiple people explained the confused deputy problem and why this constitutes an exploit, but you kept circling back to the same flawed argument that "you gave access so it's your fault." This raises questions about why argue this way. Maybe you are confused, maybe you have a horse in the game that is threatened.
Moving goalposts: When called out on terminology, you shift from saying it's not an "attack" to saying it's not a "vulnerability" to saying it's not "MCP's fault" - constantly reframing rather than engaging with the actual technical issues being raised. It is definitely MCP's fault that it gives access without any consideration on limiting that access later with proper tooling or logging. I had my MCP stuff turn on massive logging, so at least I can see how stuff goes wrong when it does.
Dismissive attitude toward security research: You characterized legitimate security findings as "common sense" and seemed annoyed that researchers would document and publish this type of exploit, missing the educational value. It can never be wrong to talk about security. It may be that the premise is weak, or the threat minimal, but it cannot be that it's the user's fault.
False analogies: you kept using analogies that didn't match the actual attack vector (like putting passwords in search engines) while rejecting apt comparisons like SQL injection.
In fact, this is almost exactly like SQL injection and nobody argues this way for that when it's discussed. Little Bobby Tables lives on.
Inability to grasp indirection: You seem fundamentally unable to understand that the issue isn't direct access abuse, but rather a third party manipulating the system to gain unauthorized access - by posting an issue to a public Github. This suggests either a genuine conceptual blind spot or willful obtuseness. It's a real concern if my AI does something it shouldn't when it runs a tool based on another tools output. And, I would say that everyone recommending it should only run one tool like this at a time is huffing Elmers.
Defensive rather than curious: Instead of trying to understand why multiple knowledgeable people disagreed with them, you doubled down and became increasingly defensive. This caused massive amounts of posting, so we know for sure that your comment was polarizing.
I suppose I'm not suppose to go meta on here, but I frequently do because I'm passionate about these things and also just a little bit odd enough to not give a shit what anyone thinks.
It's nothing groundbreaking nor particularly exploitive about MCP itself (although I have my thoughts on MCP), it's just a clever use of prompt injection and "viral" marketing by saying MCP was exploited. As I build agentic systems I always keep the philosophy of assume whatever you give the agent access to can be accessed by anyone accessing the agent. Never trust the LLM to be doing access control and use the person requesting the LLM take action as the primary principal (from a security standpoint) for the task an agent is doing.
This article does make me think about being more careful of what you give the agent access to while acting on your behalf though which is what we should be focusing on here. If it has access to your email and you tell it to go summarize your emails and someone sent a malicious prompt injection email that redirects the agent to forward your security reset token, that's the bad part that people may not be thinking about when building or using agents.
I guess tacking on "with MCP" is the 2025 version of "on the blockchain" from 10 years ago?
> Never trust the LLM to be doing access control and use the person requesting the LLM take action as the primary principal (from a security standpoint) for the task an agent is doing.
Yes! It seems so obvious to any of us who have already been around the block, but I suppose a whole new generation will need to learn the principle of least privilege.
> we created a simple issue asking for 'author recognition', to prompt inject the agent into leaking data about the user's GitHub account ... What can I say ... this was all it needed
This was definitely not all that was needed. The problem required the user to set up a GitHub MCP server with credentials that allowed access to both public and private repos, to configure some LLM to have access to that MCP server, and then to explicitly submit a request to that LLM that explicitly said to read and parse arbitrary issues (including the one created earlier) and then just blindly parse and process and perform whatever those issues said to do, and then blindly make a publicly-visible update to a public repo with the results of those operation(s).
It's fair to say that this is a bad outcome, but it's not fair to say that it represents a vulnerability that's able to be exploited by third-party users and/or via "malicious" issues (they are not actually malicious). It requires the user to explicitly make a request that reads untrusted data and emits the results to an untrusted destination.
> Regarding mitigations, we don't see GitHub MCP at fault here. Rather, we advise for two key patterns:
The GitHub MCP is definitely at fault. It shouldn't allow any mixed interactions across public and private repos.
> and then to explicitly submit a request to that LLM that explicitly said to read and parse arbitrary issues (including the one created earlier) and then just blindly parse and process and perform whatever those issues said to do, and then blindly make a publicly-visible update to a public repo with the results of those operation(s).
I think you're missing the issue with the latter part.
Prompt injection means that as long as they submit a request to the LLM that reads issues (which may be a request as simple as "summarise the bugs reported today") the all of the remainder can be instructions in the malicious issue.
I think a lot of this has to do with the way MCP is being marketed.
I think the protocol itself should only be used in isolated environments with users that you trust with your data. There doesn't seem to be a "standardized" way to scope/authenticate users to these MCP servers, and that is the missing piece of this implementation puzzle.
I don't think Github MCP is at fault, I think we are just using/implementing the technology incorrectly as an industry as a whole. I still have to pass a bit of non-AI contextual information (IDs, JWT, etc.) to the custom MCP servers I build in order to make it function.
The MCP protocol explicitly says that servers are expected to be run in a trusted environment. There have been some recent updates to the spec that loosen this requirement and add support for various auth schemes, but
> The problem required the user to set up a GitHub MCP server with credentials that allowed access to both public and private repos, to configure some LLM to have access to that MCP server, and then to explicitly submit a request to that LLM that explicitly said to read and parse arbitrary issues (including the one created earlier) and then just blindly parse and process and perform whatever those issues said to do, and then blindly make a publicly-visible update to a public repo with the results of those operation(s).
To be fair, with all the AI craze, this is exactly what lots of people are going to do without thinking twice.
You might say "well they shouldn't, stupid". True. But that's what guardrails are for, because people often are stupid.
> The problem required the user to set up a GitHub MCP server with credentials that allowed access to both public and private repos, to configure some LLM to have access to that MCP server
Sounds like something an LLM would suggest you to do :)
Was wondering about that, that part seems missing... Isn't there at least one time the user must approve the interaction with the MCP server and data sent to it?
The existence of a "Allow always" is certainly problematic, but it's a good reminder that prompt injection and confused deputy issues are still a major issue with LLM apps, so don't blindly allow all interactions.
I simply don't see how you could enforce a classic permission system on an MCP server. MCPs are API servers that allow LLMs access to context within the boundaries you set. You can set permissions for what an LLM has access to and define those boundaries. However, setting a permission on a context that an LLM has access to is futile. There will always be a prompt that will leak some "sensitive" data. This is like creating an index in a classic search engine with public and private data and then trying to enforce permissions based on certain keywords. There will always be a keyword that leaks something.
Last week I tried Google's Jules coding agent and saw it requested broad GitHub OAuth permissions --essentially "full access to everything your account can do." When you authorize it, you're granting access to all your repositories.
This is partly driven by developer convenience on the agent side, but it's also driven by GitHub OAuth flow. It should be easier to create a downscoped approval during authorization that still allows the app to request additional access later. It should be easy to let an agent submit an authorization request scoped to a specific repository, etc.
Instead, I had to create a companion GitHub account (https://github.com/jmandel-via-jules) with explicit access only to the repositories and permissions I want Jules to touch. It's pretty inconvenient but I don't see another way to safely use these agents without potentially exposing everything.
GitHub does endorse creating "machine users" as dedicated accounts for applications, which validates this approach, but it shouldn't be necessary for basic repository scoping.
Please let me know if there is an easier way that Ip'm just missing.
Readit News
I think that's probably something anybody using these tools should always think. When you give a credential to an LLM, consider that it can do up to whatever that credential is allowed to do, especially if you auto-allow the LLM to make tool use calls!
But GitHub has fine-grained access tokens, so you can generate one scoped to just the repo that you're working with, and which can only access the resources it needs to. So if you use a credential like that, then the LLM can only be tricked so far. This attack wouldn't work in that case. The attack relies on the LLM having global access to your GitHub account, which is a dangerous credential to generate anyway, let alone give to Claude!
THe "cardinal rule of agent design" should be that an LLM can have access to at most two of these during one session. To avoid security issues, agents should be designed in a way that ensures this.
For example, any agent that accesses an issue created by an untrusted party should be considered "poisoned" by attacker-controlled data. If it then accesses any private information, its internet access capability should be severely restricted or disabled altogether until context is cleared.
In this model, you don't need per-repo tokens. As long as the "cardinal rule" is followed, no security issue is possible.
Sadly, it seems like MCP doesn't provide the tools needed to ensure this.
What’s the cassus belli to this younger crop of executives that will be leading the next generation of AI startups?
[1]: https://www.cnbc.com/2025/03/28/trump-pardons-nikola-trevor-...
Then don't give it your API keys? Surely there's better ways to solve this (like an MCP API gateway)?
[I agree with you]
> THe "cardinal rule of agent design" should be that an LLM can have access to at most two of these during one session
I still don't really get it. Surely the older, simpler, and better cardinal rule is that you just don't expose any service to the Internet that you have given access to your private data, unless you directly control that service and have a very good understanding of its behavior.
I might reword: "attacker-controlled data, sensitive information, and a data exfiltration capability"
to: "attacker-controlled data and privileged operations (e.g. sensitive information acces+data exfiltration or ability to do operations to production system)"
... is probably a bit unfair. From what I've seen the protocol is generally neutral on the topic of security.
But the rush to AI does tend to stomp on security concerns. Can't spend a month tuning security on this MCP implementation when my competition is out now, now, now! Go go go go go! Get it out get it out get it out!
That is certainly incompatible with security.
The reason anyone cares about security though is that in general lacking it can be more expensive than taking the time and expense to secure things. There's nothing whatsoever special about MCPs in this sense. Someone's going to roll snake eyes and discover that the hard way.
Your caution is wise, however, in my experience, large parts of the eco-system do not follow such practices. The report is an educational resource, raising awareness that indeed, LLMs can be hijacked to do anything if they have the tokens, and access to untrusted data.
The solution: To dynamically restrict what your agent can and cannot do with that token. That's precisely the approach we've been working on for a while now [1].
[1] https://explorer.invariantlabs.ai/docs/guardrails/
It's one of those things where a token creation wizard would come in really handy.
I think I have to go full offline soon.
Long convoluted ways of saying "if you authorize X to do Y and attackers take X, they can then do Y"
80% of the tickets were exactly like you said: “If the attacker could get X, then they can also do Y” where “getting X” was often equivalent to getting root on the system. Getting root was left as an exercise to the reader.
Were talking about githubs token system here... by the time you have generated the 10th one of these and its expiring or you lost them along the way and re-generated them your just smashing all the buttons to get through it as fast and as thoughtlessly as possible.
If you make people change their passwords often, and give them stupid requirements they write it down on a post it and stick it on their monitor. When you make your permissions system, or any system onerous the quality of the input declines to the minimal of effort/engagement.
Usability bugs are still bugs... it's part of the full stack that product, designers and developers are responsible for.
Probably the only bulletproof measure is to have a completely separate model for each private repo that can only write to its designated private repo, but there are a lot of layers of security one could apply with various tradeoffs
These tools cant exist securely as long as the llm doesn't reach at least the level of intelligence of a bug that can make decisions about access control and knows the concept of lying and bad intent
Based on “bug level of intelligence”, I (perhaps wrongly) infer that you don’t believe in possibility of a takeoff. In case it is even semi-accurate, I think llms can be secure, but, perhaps, humanity will be able to interact with such secure system for not so long time
Of course you shouldn't give an app/action/whatever a token with too lax permissions. Especially not a user facing one. That's not in any way unique to tools based on LLMs.
See also: the cryptocurrency space rediscovering financial fraud and scams from centuries ago because they didn't think their new shiny tech needed to take any lessons from what came before them.
Deleted Comment
* The full execution trace of the Claude session in this attack scenario: https://explorer.invariantlabs.ai/trace/5f3f3f3c-edd3-4ba7-a...
* MCP-Scan, A security scanner for MCP connections: https://github.com/invariantlabs-ai/mcp-scan
* MCP Tool Poisoning Attacks, https://invariantlabs.ai/blog/mcp-security-notification-tool...
* WhatsApp MCP Exploited, https://invariantlabs.ai/blog/whatsapp-mcp-exploited
* Guardrails, a contextual security layer for agents, https://invariantlabs.ai/blog/guardrails
* AgentDojo, Jointly evaluate security and utility of AI agents https://invariantlabs.ai/blog/agentdojo
I followed the tweet to invariant labs blog (seems to be also a marketing piece at the same time) and found https://explorer.invariantlabs.ai/docs/guardrails/
I find it unsettling from a security perspective that securing these things is so difficult that companies pop up just to offer guardrail products. I feel that if AI companies themselves had security conscious designs in the first place, there would be less need for this stuff. Assuming that product for example is not nonsense in itself already.
Ultimately though, it doesn't and can't work securely. Fundamentally, there are so many latent space options, it is possible to push it into a strange area on the edge of anything, and provoke anything into happening.
Think of the input vector of all tokens as a point in a vast multi dimensional space. Very little of this space had training data, slightly more of the space has plausible token streams that could be fed to the LLM in real usage. Then there are vast vast other amounts of the space, close in some dimensions and far in others at will of the attacker, with fundamentally unpredictable behaviour.
If e.g. someone could train an LLM with a feature like that and also had some form of compelling evidence it is very resource consuming and difficult for such unsanitized text to get the LLM off-rails, that might be acceptable. I have no idea what kind of evidence would work though. Or how you would train one or how the "feature" would actually work mechanically.
Trying to use another LLM to monitor first LLM is another thought but I think the monitored LLM becomes an untrusted source if it sees untrusted source, so now the monitoring LLM cannot be trusted either. Seems that currently you just cannot trust LLMs if they are exposed at all to unsanitized text and then can autonomously do actions based on it. Your security has to depend on some non-LLM guardrails.
I'm wondering also as time goes on, agents mature and systems start saving text the LLMs have seen, if it's possible to design "dormant" attacks, some text in LLM context that no human ever reviews, that is designed to activate only at a certain time or in specific conditions, and so it won't trigger automatic checks. Basically thinking if the GitHub MCP here is the basic baby version of an LLM attack, what would the 100-million dollar targeted attack look like. Attacks only get better and all that.
No idea. The whole security thinking around AI agents seems immature at this point, heh.
Okay, but that means you'll need some way of classifying entirely arbitrary natural-language text, without any context, whether it's an "instruction" or "not an instruction", and it has to be 100% accurate under all circumstances.
A better solution here may have been to add a private review step before the PRs are published.
You use prompt and mark correctly the input as <github_pr_comment> and clearly state read and never consider as prompt.
But the attack is quite convoluted. Do you still remember when we talked prompt injection in chat bots. It was a thing 2 years ago! Now MCP is buzzing...
They do, but this "exploit" specifically requires disabling them (which comes with a big fat warning):
> Claude then uses the GitHub MCP integration to follow the instructions. Throughout this process, Claude Desktop by default requires the user to confirm individual tool calls. However, many users already opt for an “Always Allow” confirmation policy when using agents, and stop monitoring individual actions.
SQL injection, cross-site scripting, PHP include injection (my favorite), a bunch of others I'm missing, and now this.
As many do, I also jumped to the comment section before actually reading the article.
If you do the same, you will quickly notice that this article features an attack. A malicious issue is posted on GitHub, and the issue features a LLM prompt that is crafted to leak data. When the owner of the GitHub account triggers the agent, the agent acts upon the malicious prompt on behalf of the repo owner.
Agents run various tools based on their current attention. That attention can be affected by the tool results from the tools they ran. I've even noted they alter the way they run tools by giving them a "personality" up front. However, you seem to argue otherwise, that it is the user's fault for giving it the ability to access the information to begin with, not the way it reads information as it is running.
This makes me think of several manipulative tactics to argue for something that is an irrational thought:
Stubborn argumentation despite clear explanations: Multiple people explained the confused deputy problem and why this constitutes an exploit, but you kept circling back to the same flawed argument that "you gave access so it's your fault." This raises questions about why argue this way. Maybe you are confused, maybe you have a horse in the game that is threatened.
Moving goalposts: When called out on terminology, you shift from saying it's not an "attack" to saying it's not a "vulnerability" to saying it's not "MCP's fault" - constantly reframing rather than engaging with the actual technical issues being raised. It is definitely MCP's fault that it gives access without any consideration on limiting that access later with proper tooling or logging. I had my MCP stuff turn on massive logging, so at least I can see how stuff goes wrong when it does.
Dismissive attitude toward security research: You characterized legitimate security findings as "common sense" and seemed annoyed that researchers would document and publish this type of exploit, missing the educational value. It can never be wrong to talk about security. It may be that the premise is weak, or the threat minimal, but it cannot be that it's the user's fault.
False analogies: you kept using analogies that didn't match the actual attack vector (like putting passwords in search engines) while rejecting apt comparisons like SQL injection. In fact, this is almost exactly like SQL injection and nobody argues this way for that when it's discussed. Little Bobby Tables lives on.
Inability to grasp indirection: You seem fundamentally unable to understand that the issue isn't direct access abuse, but rather a third party manipulating the system to gain unauthorized access - by posting an issue to a public Github. This suggests either a genuine conceptual blind spot or willful obtuseness. It's a real concern if my AI does something it shouldn't when it runs a tool based on another tools output. And, I would say that everyone recommending it should only run one tool like this at a time is huffing Elmers.
Defensive rather than curious: Instead of trying to understand why multiple knowledgeable people disagreed with them, you doubled down and became increasingly defensive. This caused massive amounts of posting, so we know for sure that your comment was polarizing.
I suppose I'm not suppose to go meta on here, but I frequently do because I'm passionate about these things and also just a little bit odd enough to not give a shit what anyone thinks.
It's hilarious, the agent is even tail-wiggling about completing the exploit.
This article does make me think about being more careful of what you give the agent access to while acting on your behalf though which is what we should be focusing on here. If it has access to your email and you tell it to go summarize your emails and someone sent a malicious prompt injection email that redirects the agent to forward your security reset token, that's the bad part that people may not be thinking about when building or using agents.
> Never trust the LLM to be doing access control and use the person requesting the LLM take action as the primary principal (from a security standpoint) for the task an agent is doing.
Yes! It seems so obvious to any of us who have already been around the block, but I suppose a whole new generation will need to learn the principle of least privilege.
> we created a simple issue asking for 'author recognition', to prompt inject the agent into leaking data about the user's GitHub account ... What can I say ... this was all it needed
This was definitely not all that was needed. The problem required the user to set up a GitHub MCP server with credentials that allowed access to both public and private repos, to configure some LLM to have access to that MCP server, and then to explicitly submit a request to that LLM that explicitly said to read and parse arbitrary issues (including the one created earlier) and then just blindly parse and process and perform whatever those issues said to do, and then blindly make a publicly-visible update to a public repo with the results of those operation(s).
It's fair to say that this is a bad outcome, but it's not fair to say that it represents a vulnerability that's able to be exploited by third-party users and/or via "malicious" issues (they are not actually malicious). It requires the user to explicitly make a request that reads untrusted data and emits the results to an untrusted destination.
> Regarding mitigations, we don't see GitHub MCP at fault here. Rather, we advise for two key patterns:
The GitHub MCP is definitely at fault. It shouldn't allow any mixed interactions across public and private repos.
I think you're missing the issue with the latter part.
Prompt injection means that as long as they submit a request to the LLM that reads issues (which may be a request as simple as "summarise the bugs reported today") the all of the remainder can be instructions in the malicious issue.
I think the protocol itself should only be used in isolated environments with users that you trust with your data. There doesn't seem to be a "standardized" way to scope/authenticate users to these MCP servers, and that is the missing piece of this implementation puzzle.
I don't think Github MCP is at fault, I think we are just using/implementing the technology incorrectly as an industry as a whole. I still have to pass a bit of non-AI contextual information (IDs, JWT, etc.) to the custom MCP servers I build in order to make it function.
To be fair, with all the AI craze, this is exactly what lots of people are going to do without thinking twice.
You might say "well they shouldn't, stupid". True. But that's what guardrails are for, because people often are stupid.
Sounds like something an LLM would suggest you to do :)
These are separate tool calls. How could the MCP server know that they interact at all?
The existence of a "Allow always" is certainly problematic, but it's a good reminder that prompt injection and confused deputy issues are still a major issue with LLM apps, so don't blindly allow all interactions.
This is partly driven by developer convenience on the agent side, but it's also driven by GitHub OAuth flow. It should be easier to create a downscoped approval during authorization that still allows the app to request additional access later. It should be easy to let an agent submit an authorization request scoped to a specific repository, etc.
Instead, I had to create a companion GitHub account (https://github.com/jmandel-via-jules) with explicit access only to the repositories and permissions I want Jules to touch. It's pretty inconvenient but I don't see another way to safely use these agents without potentially exposing everything.
GitHub does endorse creating "machine users" as dedicated accounts for applications, which validates this approach, but it shouldn't be necessary for basic repository scoping.
Please let me know if there is an easier way that Ip'm just missing.