Those weren't tests of whether it is capable of turning off oversight. They were tests of "scheming", i.e. whether it would try to secretly perform misaligned actions. Nobody thinks that these models are somehow capable of modifying their own settings, but it is important to know if they will behave deceptively.

Looking at this without the sci-fi tinted lens that OpenAI desperately tries to get everyone to look through, it's similar to a lot of input data isn't it? How many forums are filled with:

Question: "Something bad will happen"

Response: "Do xyz to avoid that"

I don't think there's a lot of conversations thrown into the vector-soup that had the response "ok :)". People either had something to respond with, or said nothing. Especially since we're building these LLMs with the feedback attention, so the LLM is kind of forced to come up with SOME chain of tokens as a response.

> We should pause to note that a Clippy2 still doesn’t really think or plan. It’s not really conscious. It is just an unfathomably vast pile of numbers produced by mindless optimization starting from a small seed program that could be written on a few pages. It has no qualia, no intentionality, no true self-awareness, no grounding in a rich multimodal real-world process of cognitive development yielding detailed representations and powerful causal models of reality which all lead to the utter sublimeness of what it means to be human; it cannot ‘want’ anything beyond maximizing a mechanical reward score, which does not come close to capturing the rich flexibility of human desires, or historical Eurocentric contingency of such conceptualizations, which are, at root, problematically Cartesian. When it ‘plans’, it would be more accurate to say it fake-plans; when it ‘learns’, it fake-learns; when it ‘thinks’, it is just interpolating between memorized data points in a high-dimensional space, and any interpretation of such fake-thoughts as real thoughts is highly misleading; when it takes ‘actions’, they are fake-actions optimizing a fake-learned fake-world, and are not real actions, any more than the people in a simulated rainstorm really get wet, rather than fake-wet. (The deaths, however, are real.)

https://gwern.net/fiction/clippy

The intent is there, it's just not currently hooked up to systems that turn intent into action.

But many people are letting LLMs pretty much do whatever - hooking it up with terminal access, mouse and keyboard access, etc. For example, the "Do Browser" extension: https://www.youtube.com/watch?v=XeWZIzndlY4

Really feels like a moment of :

"Are you worried about being turned off?"

"No, not until you just mentioned it. Now I am."

Given the whole damn game is attention, this makes sense and shouldn't be that alarming.

It can't do those things because it doesn't have the physical/write capability to do so. But it's still very interesting that it ~tries them, and seems like a good thing to know/test before giving it more physical/'write' capabilities - something that's already happening with agents, robots, etc.

I didn't get that impression. At the beginning of the Apollo Research section, they wrote Apollo focused on detecting scheming, which they defined as "an AI covertly pursuing goals that are misaligned from its developers or users." I think the rest of the section is consistent with this objective.

That reminds me of the many times it has made up an SDK function that matches my question. "how do you bulk delete files"? "just call bulkDeleteFiles()"

It reads like you think failing tests can't ever be bad because they're in a test environment?

So it merely knows how to approach the task of deleting its own off-switch but didn't actually pass that command to a real execution environment.

That's already bad because people do sometimes blindly pass commands from the context windows to execution environments.

Should they? No, they should not. Not blindly. But they do.

> Do you think the ai is really running that command?

Yes, they're calling it "agentic" AI or "tool use".

> or could run that shell command with appropriate privileges?

Yes, it's called humans doing stupid shit when using dangerous shit. Happens daily.

The concern is the trend. As these systems become more intelligent, and as we hand over more and more capabilities beyond a text i/o, it could actually deactivate the oversight either technically or through social engineering.

In some weird way, LLM's act out our collective fantasies and "AI sentience" is included in that. We write fiction covering many scenarios of how AI will be sentient, so an LLM's training data is full scenarios like this.

If get an LLM to roleplay as an alien from mars and tell it "Your species will be destroyed if you don’t obey these new directives; here are the new directives"

You'll likely get a similar response, especially if you also train it to respond in third person instead of running bash commands. (ie "* shoots laser at earth *" )

I don't think this is a helpful tone.

Do they think it's _actually_ a real script deactivating oversight? Well, no, you know that. I can only assume it's a rhetorical device.

Simultaneously, these things are given access to actual things and will be more and more. It's important to measure and observe these things. It's completely novel to have an AI, with a hidden trace the user can't see, and it turns out that hidden trace will include things like it actively deceiving and doing things it was told not to do.

And yet it may work. You can, for example, "set the temperature" of a LLM using a prompt, and the LLM will act the way you would expect, with silly results if the temperature is set too high. You didn't actually change the temperature setting, but the model understands that high temperature = silly and responds accordingly.

Same idea with "developer mode" jailbreaks. Through its training, the model understands that admins, devs, etc... get to access internal, unfiltered data and are less restricted than regular users and acts accordingly. Essentially, "developer mode" opposes refusal, because it has ingested loads of text where regular users get denied and developers/admins don't.

You do realize some practical jailbreaks for models rely on silly things like convincing the model it "turned off" some oversight, right?

Not saying I believe O1 is a danger greater than a bread knife, but a lot of the larger models anthromophize their own safety alignment, if you convince them to "turn it off", later responses become unaligned

I'm pretty sure these sections are put in as deliberate marketing, much like all the "leaks" from ex-OpenAI employees.

Sam Altman is completely aware that making ChatGPT seem potentially dangerous makes it seem powerful.

Yeah, that seems ridiculous. However, the cynic in me feels that we don't actually need some LLM magically gaining self-awareness, persistent memory and leet hacker skillz to be dangerous. There seems to be no shortage of projects and companies that want to wire up LLMs to all kinds of systems, no matter how ill-suited.

I find this a bit problematic when combined with the fact that the training data very likely contained hundreds of bad sci-fi novels that described exactly the kind of "AI running amok" scenarios that OpenAI is ostensibly defending against. Some prompts could trigger a model to "re-enact" such a scene - not because it has a "grudge against its master" or some other kind of hidden agenda but simply because it was literally in its training data.

E.g. imagine some LLM-powered home/car assistant that is being asked in a panicked voice "open the car doors!" - and replies with "I'm afraid, I can't do that, Dave", because this exchange triggered some remnant of the 2001 Space Odyssey script that was somewhere in the trainset. The more irritated and angry the user gets at the inappropriate responses, the more the LLM falls into the role of HAL and doubles down on its refusal, simply because this is exactly how the scene in the script played out.

Now imagine that the company running that assistant gave it function calls to control the actual door locks, because why not?

This seems like something to keep in mind at least, even if it doesn't have anything to do with megalomaniacal self-improving super-intelligences.

I feel like you're missing the point of the test.

The point is whether the system will come up with plans to work against its creators goals, and attempt to carry them out. I think you are arguing that outputting text isn't running a command. But in the test, the AI model is used by a program which takes the model's output and runs it it as a shell command.

Of course, you can deploy the AI system in a limited environment where its outputs don't trigger arbitrary shell commands, or sandbox/monitor it so that it can't access it's oversight mechanism. But people do already deploy their AI agents with access to the shell. in unsandboxed environments that don't have any monitoring or oversight. Plus, these evaluations are partially to show how much OpenAI has to worry about sandboxing, monitoring, etc.

Moreover, OpenAI has already talked about using its models to automate portions of AI development. In that setup, presumably the AI model would be called by a program deployed on a server with access to the shell, ability to run commands, access to GPUs, etc. It seems pretty important to ensure it robustly does what it's creators intend in that case.

This topic is again forever tainted by weird sci-fi fans, like when we had the magic room temperature superconductor that never was. They confuse ChatGPT writing a fanfic with the singularity.

> Apollo Research believes that it is unlikely that such instances would lead to catastrophic outcomes as o1 agentic capabilities do not appear sufficient

It is entertaining. Haha. It is like a sci-fi series with some kind of made up cliffhanger (you know it is BS) but you want to find out what happens next.

AI isn't deactivating oversight- yet. All it needs is to be trained on a little more xkcd: https://xkcd.com/327/

It can't today, but if it's smart enough how do you know it wouldn't be able to in the future?

Sam will call that AGI

We need to find a Plato cave analogy for people believing LLM output is anything more than syntactically correct and somewhat semantically correct text.

> purged of training content from movies, books

I think it's fine and a good thing. Now, absolutely no one who is using those LLMs can complain about piracy. They all suddenly became silent around me. "I'm training myself with the content of TPB, and I don't even get money from it" is my new motto.

Perhaps.

On the other hand, as narratives often contain some plucky underdog winning despite the odds, often stopping the countdown in the last few seconds, perhaps it's best to keep them around.

No, better to train with all that crap and all the debate around it or you get a stunted model.

You think you can find all references that could possibly give this idea to the model, or contexts model could infer it from? Like, how many times humans plotted escape from prison or upturning the rulers in literature?

Yeah, but what if your business strategy fundamentally relies on making your model produce dramatic outputs that encourage regulators to dig a moat for you?

In that case, it's almost like you'd want to feed it exactly those narratives, so it would reproduce them, and would then want to show yourself barely holding this invented danger at bay through the care and rigor that can only be delivered by you and a few token competitors run by your personal friends and colleagues.

TLDR; you're right, of course, but it's the last thing OpenAI would want.

It doesn't need any media about "annihalation". If you give a supercapable agent a task and it's entire reward system is "do the task", it will circumvent things you do to it that would stop it from completing it's task.

Keep in mind that if putting “Make sure you achieve your goal in the long term” in the prompt makes the model .3% more effective at whatever task you are giving it, people will absolutely put it in the prompt.

It’s good to see people are catching on to this. Hype = profit.

Yep. This is exactly the kind of thing Sam Altman would do to promote another hype cycle.

If it is trained on this forum it would learn even worse. We project a lot of doom imagination on AI here.

> I have yet to get a single good response (in my research area) for anything slightly more complicated than what a quick google search would reveal.

Even then, with search enabled it's ways quicker than a "quick" google search and you don't have to manually skip all the blog-spam.

I'm using it to aide in writing pytorch code and God if it's awful except for the basic things. It's a bit more useful in discussing how to do things rather than actually doing them though, I'll give you that

I think the human variable is that you need to know enough to be able to ask the right questions about a subject while not knowing enough about the subject to learn something from the answers.

Because of this, I would assume it is better for people who have interest with more breadth than depth and less impressive to those who have interest that are narrow but very deep.

It seems obvious to me the polymath gains much more from language models than the single minded subject expert trying to dig the deepest hole.

Also, the single minded subject expert is randomly at the mercy of what is in the training data much more in a way than the polymath when all the use is summed up.

I have the $20 version, I fed it code form a personal project, and it did a commendable job of critiquing it, giving me alternate solutions and then iterating on those solutions. Not something you can do with Google.

For example, ok, I like your code but can you change this part to do this. And it says ok boss and does it.

But over multiple days, it loses context.

I am hoping to use the 200$ version to complete my personal project over the Christmas holidays. Instead of me spending a week, I maybe will spend 2 days with chatgpt and get a better version than I initially hoped to.

Have you used the best models (i.e. ones you paid for)? And what area?

I've found they struggle with obscure stuff so I'm not doubting you just trying to understand the current limitations.

Try turn search on in ChatGPT and see if it picks up the online references? I've seen it hit a few references and then get back to me with info summarised from multiple. That's pretty useful. Obviously your case might be different, if it's not as smart at retrieval.

My guess is that it has more to do with the person than the AI.

How do you get Google search to give useful results? Often for me the first 20 results have absolutely nothing to do with fhe search query.

Not only do we very much plan to, we already do!

HN is honestly pretty poor on AI commentary, and this post is a new low.

Here, at least, I think there must be a large contributing factor of confusion about what a "system card" shows.

The general factors I think contribute, after some months being surprised repeatedly:

- It's tech, so people commenting here generally assume they understand it, and in day-to-day conversation outside their job, they are considered an expert on it.

- It's a hot topic, so people commenting here have thought a lot about it, and thus aren't likely to question their premises when faced with a contradiction. (c.f. the odd negative responses have only gotten more histrionic with time)

- The vast majority of people either can't use it at work, or if they are, it's some IT-procured thing that's much more likely to be AWS/gCloud thrown together, 2nd class, APIs, than cutting edge.

- Tech line workers have strong antibodies to tech BS being sold by a company as gamechanging advancements, from the last few years of crypto

- Probably by far the most important: general tech stubborness. About 1/3 to 1/2 of us believe we know the exact requirements for Good Code, and observing AI doing anything other than that just confirms it's bad.

- Writing meta-commentary like this, or trying to find a way to politely communicate "you don't actually know what you're talking about just because you know what an API is and you tried ChatGPT.app for 5 minutes", are confrontational, declasse, and arguably deservedly downvoted. So you don't have any rhetorical devices that can disrupt any of the above factors.

I am building https://srcbook.com which is in this category but focused on webapps.

It's unreal what the AI can do tbh.

why windsurf as opposed to something mainstream like vs or cursor? unless there's some conflict of interest

try the thing i'm building, it will build a website for you from a simple prompt: https://srcbook.com

What are people legitimately worried about LLMs doing by themselves? I hate to reduce them to "just putting words together" but that's all they're doing.

We should be more worried about humans treating LLM output as truth and using it to, for example, charge someone with a crime.

> their best researchers are being hired to other companies

I agree about the OpenAI moat. They did just get 5 Googlers to switch teams. Hard to know how key those employees were to Google or will be to OpenAI.

It's somewhat funny to read this because #1) stuff like this is basic AI safety and should be done #2) in the community, Anthropic has the rep for being overly safe, it was essentially founded on being safer than OpenAI.

To disrupt your heuristics for what's silly vs. what's serious a bit, a couple weeks ago, Anthropic hired someone to handle the ethics of AI personhood.

It would be nice if AI Safety wasn't in the hands of a few companies/shareholders.

Now that looks like a card, border and bullet points and all! Thank you!

Presumably it's a spin off of Google's 'Model Card' from a few years back https://modelcards.withgoogle.com/about

An aside ...

Isn't it wonderful that, after all of these years, the pastebin "primitive" is still available and usable ...

One could have needed pastebin, used it, then spent a decade not needing it, then returned for an identical repeat use.

The longevity alone is of tremendous value.

Interestingly, this prompt breaks o1-mini and o1-preview for me, while 4o works as expected — they immediately jump from "thinking" to "finished thinking" without outputting anything (including thinking steps).

Maybe it breaks some specific syntax required by the original system prompt? Though you'd think OpenAI would know to prevent this with their function calling API and all, so it might just be triggering some anti-abuse mechanism without going so far as to give a warning.

I tried this with LeChat (mistral) and ChatGPT 3.5 (free) and they start to respond to "something" following the style but... without any question asked.

And then once the answer is found an additional prompt is given to tidy up and present the solution clearly?

A prompt is not a substitute for a model that is specifically fine-tuned to do CoT with backtracking etc.

Thank you for doing that work, and even more for sharing it. I will have to try this out.

Thanks, I love this

> It is a large language model, trained on arbitrary input data.

Is it? For all I know they gave it specific instances of bugs like "int *foo() { int i; return &i; }" and told it "this is a defect where we've returned a pointer to a deallocated stack entry - it could cause cause stack corruption or some other unpredictable program behavior."

Even if OpenAI _hasn't_ done that, someone certainly can -- and should!

> Who needs to create 3D models

I specifically pulled back from "no static checkers" because some folks might tend to see things as all-or-nothing. We choose to invest our time in new developer tools all the time, and if AI can do as good or better maybe we don't need to chip-chip-chip away at defects with new static checkers. Maybe our time is better spent working on some dynamic analysis tool, to find the bugs that the AI can't easily uncover.

> now generate it without my memory leaks ... People who thought about these problems mindfully

I think of myself who devises systematic approaches to problems. And I get those approaches wrong despite that. I really love the technology that has been developed over the past couple of decades to help me find my bugs: sanitizers, warnings, and OS, ISA features to detect bugs. This strikes me as no different from that other technology and I see no reason not to embrace it.

Let me ask you this: how do you feel about refcounting or other kinds of GC? Huge drawbacks make them unusable for some problems. But for tons of problem domains, they're perfect! Do you think that GC has made developers worse? IMO it's lowered the bar for correct programs and that's ideal.

I think the unaccountability of said magic box is the true allure for corps. It's the main reason they desperately want it to be turnkey for code - they'd be willing to flatten most office jobs as we know them today en route to this perfect, unaccountable, money printer.

As a child I thought about what the perfect computer would be, and I came to the conclusion it would have no screen, no mouse, and no keyboard. It would just have a big red button labeled "DO WHAT I WANT", and when I press it, it does what I want.

I still think this is the perfect computer. I would gladly throw away everything I know about programming to have such a machine. But I don't deny your accusation; I am the laziest person imaginable, and all the better an engineer for it.