skaul (u/skaul) - Readit News

skaul commented on Comet AI browser can get prompt injected from any site, drain your bank account twitter.com/zack_overflow... · Posted by u/helloplanets

simonw · 2 days ago

Even if you know the source of the text before you feed it to the model you still need to solve the problem of how to send untrusted text from a user through a model without that untrusted text being able to trigger additional tool calls or actions.

The most credible pattern I've seen for that comes from the DeepMind CaMeL paper - I would love to see a browser agent that robustly implemented those ideas: https://simonwillison.net/2025/Apr/11/camel/

skaul · 2 days ago

> Even if you know the source of the text before you feed it to the model you still need to solve the problem of how to send untrusted text from a user through a model without that untrusted text being able to trigger additional tool calls or actions.

We're exploring taking the action plan that a reasoning model (which sees both trusted and untrusted text) comes up with and passing it to a second model, which doesn't see the untrusted text and which then evaluates it.

> The most credible pattern I've seen for that comes from the DeepMind CaMeL paper

Yeah we're aware of the CaMeL paper and are looking into it, but it's definitely challenging from an implementation pov.

Also, I see that we said "The browser should clearly separate the user’s instructions from the website’s contents when sending them as context to the model" in the blog post. That should have been "backend", not "model". Agreed that once you feed both trusted and untrusted tokens into the LLM the output must be considered unsafe.

skaul commented on Comet AI browser can get prompt injected from any site, drain your bank account twitter.com/zack_overflow... · Posted by u/helloplanets

waterproof · 3 days ago

Isn't there a situation where the agentic browser, acting correctly on behalf of the user, needs to send Bitcoin or buy plane tickets? Isn't that flexibility kind of the whole point of the system? If so, I don't see what you get by distinguishing between agentic and no agentic browsing.

Bad actors will now be working to scam users' LLMs rather than the users themselves. You can use more LLMs to monitor the LLMs and try and protect them, but it's turtles all the way down.

The difference: when someone loses their $$$, they're not a fool for falling for some Nigerian Prince wire scam themselves, they're just a fool for using your browser.

Or am I missing something?

skaul · 3 days ago

You're right that if the user logs into a sensitive website, the "isolated browsing" mitigation stops helping. We don't want the user to accidentally end up in that state though. Separately, I can also imagine use-cases for agentic browsing where the user doesn't have to be logged into sensitive websites. Summarizing Hacker News front page, for one.

skaul commented on Comet AI browser can get prompt injected from any site, drain your bank account twitter.com/zack_overflow... · Posted by u/helloplanets

simonw · 3 days ago

"But we also need guarantees at other layers, like distinguishing web contents from user instructions"

How do you intend to do that?

In the three years I've spent researching and writing about prompt injection attacks I haven't seen a single credible technique from anyone that can distinguish content from instructions.

If you can solve that you'll have solved the entire class of prompt injection attacks!

skaul · 3 days ago

> I haven't seen a single credible technique from anyone that can distinguish content from instructions

You specifically mean that it's ~impossible to distinguish between content and instructions ONCE it is fed to the model, right? I agree with that. I was talking about a prior step, at the browser level. At the point that the query is sent to the backend, the browser would be able to distinguish between web contents and user prompt. This is useful for checking user-alignment of the output of the reasoning model (keeping in mind that the moment you feed in untrusted text into a model all bets are off).

We're actively thinking and working on this, so will have more to announce soon, but this discussion is useful!

skaul commented on Comet AI browser can get prompt injected from any site, drain your bank account twitter.com/zack_overflow... · Posted by u/helloplanets

cowboylowrez · 3 days ago

sure sure, except llms. I mean its valid and all bringing up tried and true maxims that we all should know regarding software, but whens the last time the ssl guys were happy with a fix that "has a chance of working, but a chance of not working."

defense in depth is to prevent one layer failure from getting to the next, you know, exploit chains etc. Failure in a layer is a failure, not statistically expected behavior. we fix bugs. what we need to do is treat llms as COMPLETELY UNTRUSTED user input as has been pointed out here and elsewhere time and again.

you reply to me like I need to be lectured, so consider me a dumb student in your security class. what am I missing here?

skaul · 3 days ago

> you reply to me like I need to be lectured

That's not my intention! Just stating how we're thinking about this.

> defense in depth is to prevent one layer failure from getting to the next

We think a separate model can help with one layer of this: checking if the planner model's actions are aligned with the user's request. But we also need guarantees at other layers, like distinguishing web contents from user instructions, or locking down what tools the model has access to in what context. Fundamentally, though, like we said in the blog post:

"The attack we developed shows that traditional Web security assumptions don’t hold for agentic AI, and that we need new security and privacy architectures for agentic browsing."

skaul commented on Comet AI browser can get prompt injected from any site, drain your bank account twitter.com/zack_overflow... · Posted by u/helloplanets

ryanjshaw · 3 days ago

Maybe the article was updated but right now it says “The browser should isolate agentic browsing from regular browsing”

skaul · 3 days ago

That was in the blog from the starting, and it's also the most important mitigation we identified immediately when starting to think about building agentic AI into the browser. Isolating agentic browsing while still enabling important use-cases (which is why users want to use agentic browsing in the first place) is the hard part, which is presumably why many browsers are just rolling out agentic capabilities in regular browsing.

skaul commented on Comet AI browser can get prompt injected from any site, drain your bank account twitter.com/zack_overflow... · Posted by u/helloplanets

cowboylowrez · 3 days ago

what you're saying is that the described step, "model alignment" is necessary even though it will fail a percentage of the time. whenever I see something that is "necessary" but doesn't have like a dozen 9's for reliability against failure or something well lets make that not necessary then. whadya say?

skaul · 3 days ago

That's not how defense-in-depth works. If a security mitigation catches 90% of the "easy" attacks, that's worth doing, especially when trying to give users an extremely powerful capability. It just shouldn't be the only security measure you're taking.

skaul commented on Comet AI browser can get prompt injected from any site, drain your bank account twitter.com/zack_overflow... · Posted by u/helloplanets

ec109685 · 3 days ago

It’s obviously fundamentally unsafe when Google, OpenAI and Anthropic haven’t released the same feature and instead use a locked down VM with no cookies to browse the web.

LLM within a browser that can view data across tabs is the ultimate “lethal trifecta”.

Earlier discussion: https://news.ycombinator.com/item?id=44847933

It’s interesting that in Brave’s post describing this exploit, they didn’t reach the fundamental conclusion this is a bad idea: https://brave.com/blog/comet-prompt-injection/

Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough. The only good mitigation they mention is that the agent should drop privileges, but it’s just as easy to hit an attacker controlled image url to leak data as it is to send an email.

skaul · 3 days ago

(I lead privacy at Brave and am one of the authors)

> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.

No, we never claimed or believe that those will be enough. Those are just easy things that browser vendors should be doing, and would have prevented this simple attack. These are necessary, not sufficient.

skaul commented on Brave blocks Microsoft Recall by default brave.com/privacy-updates... · Posted by u/XzetaU8

ritenuto · a month ago

Interestingly, the linked Recall docs[1] mention a way to filter apps and/or websites from being saved; however:

> This setting applies only to Enterprise and Education editions of Windows.

That limitation looks extremely impractical.

[1]: https://learn.microsoft.com/en-us/windows/client-management/...

skaul · a month ago

(disclaimer: I lead privacy at Brave and wrote the article)

See the blog post for how we implemented this: https://brave.com/privacy-updates/35-block-recall/#how-we-im.... We took Recall's guidance for web browsers and extended it to apply for ALL windows, not just Private Browsing: https://learn.microsoft.com/en-us/windows/ai/recall/recall-w...

skaul commented on Brave blocks Microsoft Recall by default brave.com/privacy-updates... · Posted by u/XzetaU8

Yolopix · a month ago

I'm tired of all these apps using Recall as a lazy way to create pointless "privacy improving" features. This is pure marketing and there is absolutely no actual intention of improving user privacy.

As far as I know, Recall has never been enabled by default on any Windows-PC, even the new "Copilot+ PCs", so this should not be a concern as users have to explicitely opt-in to enable this privacy-invading feature.

First it was Signal which pretended being "forced" to create such a feature. I love Signal but I found this absolutely ridiculous.

Preventing a Window to be seen by other programs has the side-effect of making it completely invisible when using Windows remotely with tools such as Sunshine. How am I supposed to use Brave or Signal if the setting to disable this feature is not accessible because I can't even see the settings screen first?

HN really loves making Microsoft (especially Windows) appear even worse as it already is...

skaul · a month ago

(disclaimer: I lead privacy at Brave and wrote the article)

> How am I supposed to use Brave or Signal if the setting to disable this feature is not accessible because I can't even see the settings screen first?

Brave's implementation shouldn't block screen readers or screenshot tools. It only blocks Recall. See the blog post: https://brave.com/privacy-updates/35-block-recall#disabling-...