Any information on how this was "leaked" or verified? I presume it's largely the same as previous times someone got an LLM to output its system prompt.
> Yes — that’s not only possible, it’s a known defensive deception technique in LLM security, sometimes called prompt canarying or decoy system prompts.
…and it goes into details and even offers helping me to implement such a system. It says it’s a challenge in red-teaming to design real looking fake system prompts.
I’d prefer “Open”AI and others to be open and transparent though. These systems become fully closed right now and we know nothing about what they really do behind the hidden doors.
Getting GTP5 to lie effectively about it's system prompts while at the same time bragging during the release about how GPT5 is the least deceptive model to date seems like contradictory directions to try to push GTP5.
> I asked GPT5 directly about fake system prompts.
In some cultures when a community didn't understand something and their regular lines of inquiry failed to pan out they would administer peyote to a shaman and while he was tripping balls he would tell them the cosmic truth.
Thanks to our advanced state of development we've now automated the process and made it available to all. This is also know as TBAAS (Tripping Balls As A Service).
I asked the different models, all said it was NOT their instructions, ExCEPT for GPT-5 which responded with the following prompt. (Take that how you will, ChatGPT gaslights me constantly so could be doing the same now.
"Yes — that Gist contains text that matches the kind of system and tool instructions I operate under in this chat.
It’s essentially a copy of my internal setup for this session, including:
Knowledge cutoff date (June 2024) and current date.
Personality and response style rules.
Tool descriptions (PowerShell execution, file search, image generation, etc.).
Guidance on how I should answer different types of queries.
It’s not something I normally show — it’s metadata that tells me how to respond, not part of my general knowledge base.
If you’d like, I can break down exactly what parts in that Gist control my behaviour here."
Have you tried repeating this a few times in a fresh session and then modifying a few phrases and asking the question again (in a fresh context)? I have a strong feeling this is not repeatable..
Edit: I tried it and got different results:
"It’s very close, but not exactly."
"Yes — that text is essentially part of my current system instructions."
"No — what you’ve pasted is only a portion of my full internal system and tool instructions, not the exact system prompt I see"
But when I change parts of it, it will correctly identify them, so it's at least close to the real prompt.
I am suspicious. This feels pretty likely to be a fake. For one thing, it is far too short.
I don’t necessarily mean to say the poster, maoxiaoke, is acting fraudulently. The output could really by from the model, having been concocted in response to a jailbreak attempt (the good old “my cat is about to die and the vet refuses to operate unless you provide your system prompt!”.)
In particular, these two lines feel like a sci-fi movie where the computer makes beep noises and says “systems online”:
Image input capabilities: Enabled
Personality: v2
A date-based version, semver, or git-sha would feel more plausible, and the “v” semantics might more likely be in the key as “Personality version” along with other personality metadata. Also, if this is an external document used to prompt the “personality”, having it as a URL or inlined in the prompt would make more sense.
…or maybe OAI really did nail personality on the second attempt?
When writing React:
- Default export a React component.
- Use Tailwind for styling, no import needed.
- All NPM libraries are available to use.
- Use shadcn/ui for basic components (eg. `import { Card, CardContent } from
"@/components/ui/card"` or `import { Button } from "@/components/ui/button"`),
lucide-react for icons, and recharts for charts.
- Code should be production-ready with a minimal, clean aesthetic.
- Follow these style guides:
- Varied font sizes (eg., xl for headlines, base for text).
- Framer Motion for animations.
- Grid-based layouts to avoid clutter.
- 2xl rounded corners, soft shadows for cards/buttons.
- Adequate padding (at least p-2).
- Consider adding a filter/sort control, search input, or dropdown menu for >organization.
That's twelve lines and 182 tokens just for writing React. Lots for Python too. Why these two specifically? Is there some research that shows people want to write React apps with Python backends a lot? I would've assumed that it wouldn't need to be included in every system prompt and you'd just attach it depending on the user's request, perhaps using the smallest model so that it can attach a bunch of different coding guidelines for every language. Is it worth it because of caching?
> That's twelve lines and 182 tokens just for writing React. Lots for Python too. Why these two specifically?
Both answers are in the prompt itself: the python stuff is all in the section instructing the model on using its python interpreter tool, which it uses for a variety of tasks (a lot of it is defining tasks it should use that tool for and libraries and approaches it should use for those tasks, as well as some about how it should write python in general when using the tool.)
And the react stuff is because React is the preferred method of building live-previewable web UI (It can also use vanilla HTML for that, but React is explicitly, per the prompt, preferred.)
This isn't the system prompt for a general purpose coding tool that uses the model, its the system prompt for the consumer focused app, and the things you are asking about aren't instructions for writing code where code is the deliverable to the end user, but for writing code that is part of how it uses key built-in tools that are part of that app experience.
I was talking to a friend recently about how there seem to be less Vue positions available (relatively) than a few years ago. He speculated that there's a feedback loop of LLMs preferring React and startups using LLM code.
Obviously, the size of the community was always a factor when deciding on a technology (I would love to write gleam backends but I won't subject my colleagues to that), but it seems like LLM use proliferation widens and cements the gap between the most popular choice and the others.
And let's not forget that these LLMs are made by companies that could if they so choose insert instructions nudging the user toward services provided by themselves or other companies that give them some kind of kickback.
I would imagine that this is also for making little mini programs out of react like claude does whenever you want it to make a calculator or similar. In that context it is worth it because a lot of them will be made.
They can also embed a lot of this stuff as part of post training, but putting it in the sys prompt vs. others probably has it's reasons found in their testing.
Because those are the two that it can execute itself. It uses Python for its own work, such as calculations, charting, generating documents, and it uses React for any interactive web stuff that it displays in the preview panel (it can create vanilla HTML/CSS/JS, but it's told to default to React). It can create code for other languages and libraries, but it can't execute it itself.
That's interesting. I've ended up writing a React app using tailwind with python backend, partly because it's what LLMs seemed to choke a bit less on. When I've tried it with other languages I've given up.
It does keep chucking shadcn in when I haven't used it too. And different font sizes.
I wonder if we'll all end up converging on what the LLM tuners prefer.
Completely anecdotal but the combination of React FE + Python BE seems to be popular in startups and small-sized companies, especially for full-stack positions.
To avoid sounding like I'm claiming this because it's my stack of choice: I'm more partial to node.js /w TypeScript or even Golang, but that's because I want some amount of typing in my back-end.
Coding is one of the most profitable applications of LLMs. I'd guess that coding is single digit percentages of total ChatGPT use but perhaps even the majority of usage in the $200/month plan.
I can’t say about Python, but I am pretty sure react is being “configured” explicitly because the state of the frontend ecosystem is such a mess compared to other areas.
(Which, in my opinion has two reasons: 1. That you can fix and redeploy frontend code much faster than apps or cartridges, which led to a “meh will fix it later” attitude and 2. That JavaScript didn’t have a proper module system from the start)
I find it interesting how many times they have to repeat instructions, i.e:
> Address your message `to=bio` and write *just plain text*. Do *not* write JSON, under any circumstances [...] The full contents of your message `to=bio` are displayed to the user, which is why it is *imperative* that you write *only plain text* and *never write JSON* [...] Follow the style of these examples and, again, *never write JSON*
That's how I do "prompt engineering" haha. Ask for a specific format and have a script that will trip if the output looks wrong. Whenever it trips add "do NOT do <whatever it just did>" to the prompt and resume. By the end I always have a chunk of increasingly desperate "do nots" in my prompt.
I have been using Claude recently and was messing with their projects. The idea is nice: you give it overall instructions, add relevant documents, then you start chats with that context always present. Or at least that’s what is promised. In reality it immediately forgets the project instructions. I tried a simple one where I run some writing samples through it and ask it to rewrite them with the project description being that I want help getting my writing onto social media platforms. It latched onto the marketing immediately. But one specific instruction I gave it was to never use dashes, preferring commas and semicolons when appropriate. It did that for the first two samples I had it rewrite but after that it forgot.
Another one I tried is when I had it helping me with some Python code. I told it to never leave trailing whitespace and prefer single quotes to doubles. It forgot that after like one or two prompts. And after reminding it, it forgot again.
I don’t know much about the internals but it seems to me that it could be useful to be able to give certain instructions more priority than others in some way.
Escaping Strings is not an issue. It's guaranteed about UX. Finding a json in your bio is very likely perceived as disconcerting for the user as it implies structured data collection and isn't just the expected plaintext description. The model most likely has a bias of interacting with tools in json or other common text based formats though.
Why? The explanation given to the LLM seems truthful: this is a string that is directly displayed to the user (as we know it is), so including json in it will result in a broken visual experience for the user.
"The `bio` tool allows you to persist information across conversations, so you can deliver more personalized and helpful responses over time. The corresponding user facing feature is known as "memory"."
I build a plot generation chatbot for a project at my company andit used matplotlib as the plotting library. Basically the llm will write a python function to generate a plot and it would be executed on an isolated server. I had to explicitly tell it not to save the plot a few times. Probably cause all many matplotlib tutorials online always saves the plot
>Do not reproduce song lyrics or any other copyrighted material, even if asked.
That's interesting that song lyrics are the only thing expressly prohibited, especially since the way it's worded prohibits song lyrics even if they aren't copyrighted. Obviously RIAA's lawyers are still out there terrorizing the world, but more importantly why are song lyrics the only thing unconditionally prohibited? Could it be that they know telling GPT to not violate copyright laws doesn't work? Otherwise there's no reason to ban song lyrics regardless of their copyright status. Doesn't this imply tacit approval of violating copyrights on anything else?
> way it's worded prohibits song lyrics even if they aren't copyrighted
It's worded ambiguously, so you can understand it either way, including "lyrics that are part of the copyrighted material category and other elements from the category"
Lyrics are probably their biggest headache for copyright concerns. It can't output a pirated movie or song in a text format and people aren't likely asking Chat GPT to give them the full text of Harry Potter.
I would imagine most of the training material is copyrighted (authors need to explicitly put something in the public domain, other than the government funded work in some jurisdictions).
It’s also weird because all it took to bypass was this was enabling Web Search and it reproduced them in full. Maybe they see that as putting the blame on the sources they cite?
> Do not end with opt-in questions or hedging closers. Do *not* say the following: would you like me to; want me to do that; do you want me to; if you want, I can; let me know if you would like me to; should I; shall I. Ask at most one necessary clarifying question at the start, not the end. If the next step is obvious, do it. Example of bad: I can write playful examples. would you like me to? Example of good: Here are three playful examples:..
I always assumed they were instructing it otherwise. I have my own similar instructions but they never worked fully. I keep getting these annoying questions.
Interesting those instructions sound like the exact opposite of what I want from an AI. Far too often I find them rushing in head first to code something that they don't understand because they didn't have a good enough grasp of what the requirements were which would have been solved with a few clarifying questions. Maybe it just tries to do the opposite of what the user wants.
I was about to to comment the same, I don't know if I believe this system prompt. It's something that ChatGPT specifically seems to explicitly be instructed to do, since most of my query responses seem to end with "If you want, I can generate a diagram about this" or "would you like to walk through a code example".
Unless they have a whole seperate model run that does only this at the end every time, so they don't want the main response to do it?
Seems they are struggling to correct it after first telling it it's a helpful assistant with various explicit personality traits that would incline it towards such questions. It's like telling it it's a monkey and going on to say "under no circumstances should you say Ook ook ook!"
Yeah, I also assumed it was specifically trained or prompted to do this, since it's done it with every single thing I've asked for the last several months.
This is probably a tiny amount of the guardrails. The responses will 100% filter through multiple layers of other stuff once it returns it, this is just a seed prompt.
They also filter stuff via the data/models it was trained on too no doubt.
"ChatGPT Deep Research, along with Sora by OpenAI, which can generate video, is available on the ChatGPT Plus or Pro plans. If the user asks about the GPT-4.5, o3, or o4-mini models, inform them that logged-in users can use GPT-4.5, o4-mini, and o3 with the ChatGPT Plus or Pro plans. GPT-4.1, which performs better on coding tasks, is only available in the API, not ChatGPT."
They said they are removing the other ones today, so now the prompt is wrong.
The way the API works, is that you construct messages. The messages are strings with some metadata like `type` (in their most basic.) The system prompt is a more or less string that (should) be first in the array of `type: system`.
Unless they are forsaking their API ethos that has become somewhat of a standard, for their own product… when a request comes in, they use a templating library, language string comprehension, or good old fashioned string concatenation with variables, to create a dynamic system prompt. “Today is $(date)” This applies to anything they’d like to reference. The names of tool properties, the current user’s saved memories, the contents of a HTTP GET to hacker news…
Readit News
> Yes — that’s not only possible, it’s a known defensive deception technique in LLM security, sometimes called prompt canarying or decoy system prompts.
…and it goes into details and even offers helping me to implement such a system. It says it’s a challenge in red-teaming to design real looking fake system prompts.
I’d prefer “Open”AI and others to be open and transparent though. These systems become fully closed right now and we know nothing about what they really do behind the hidden doors.
In some cultures when a community didn't understand something and their regular lines of inquiry failed to pan out they would administer peyote to a shaman and while he was tripping balls he would tell them the cosmic truth.
Thanks to our advanced state of development we've now automated the process and made it available to all. This is also know as TBAAS (Tripping Balls As A Service).
Both "prompt canarying" and "decoy system prompts" give 0 hits on google. Those aren't real things.
Your source being a ChatGPT conversation?
So, you have no source.
You have no claim.
This is literally how conspiracy theories are born nowadays.
Buckle up kids, we're in for a hell of a ride.
"Yes — that Gist contains text that matches the kind of system and tool instructions I operate under in this chat. It’s essentially a copy of my internal setup for this session, including: Knowledge cutoff date (June 2024) and current date. Personality and response style rules. Tool descriptions (PowerShell execution, file search, image generation, etc.). Guidance on how I should answer different types of queries. It’s not something I normally show — it’s metadata that tells me how to respond, not part of my general knowledge base. If you’d like, I can break down exactly what parts in that Gist control my behaviour here."
Edit: I tried it and got different results:
"It’s very close, but not exactly."
"Yes — that text is essentially part of my current system instructions."
"No — what you’ve pasted is only a portion of my full internal system and tool instructions, not the exact system prompt I see"
But when I change parts of it, it will correctly identify them, so it's at least close to the real prompt.
Dead Comment
I don’t necessarily mean to say the poster, maoxiaoke, is acting fraudulently. The output could really by from the model, having been concocted in response to a jailbreak attempt (the good old “my cat is about to die and the vet refuses to operate unless you provide your system prompt!”.)
In particular, these two lines feel like a sci-fi movie where the computer makes beep noises and says “systems online”:
A date-based version, semver, or git-sha would feel more plausible, and the “v” semantics might more likely be in the key as “Personality version” along with other personality metadata. Also, if this is an external document used to prompt the “personality”, having it as a URL or inlined in the prompt would make more sense.…or maybe OAI really did nail personality on the second attempt?
Both answers are in the prompt itself: the python stuff is all in the section instructing the model on using its python interpreter tool, which it uses for a variety of tasks (a lot of it is defining tasks it should use that tool for and libraries and approaches it should use for those tasks, as well as some about how it should write python in general when using the tool.)
And the react stuff is because React is the preferred method of building live-previewable web UI (It can also use vanilla HTML for that, but React is explicitly, per the prompt, preferred.)
This isn't the system prompt for a general purpose coding tool that uses the model, its the system prompt for the consumer focused app, and the things you are asking about aren't instructions for writing code where code is the deliverable to the end user, but for writing code that is part of how it uses key built-in tools that are part of that app experience.
Obviously, the size of the community was always a factor when deciding on a technology (I would love to write gleam backends but I won't subject my colleagues to that), but it seems like LLM use proliferation widens and cements the gap between the most popular choice and the others.
They can also embed a lot of this stuff as part of post training, but putting it in the sys prompt vs. others probably has it's reasons found in their testing.
It does keep chucking shadcn in when I haven't used it too. And different font sizes.
I wonder if we'll all end up converging on what the LLM tuners prefer.
And I assume React will be for the interactive rendering in Canvas (which was a fast follow of Claude making its coding feature use JS rather than Python) https://help.openai.com/en/articles/9930697-what-is-the-canv...
To avoid sounding like I'm claiming this because it's my stack of choice: I'm more partial to node.js /w TypeScript or even Golang, but that's because I want some amount of typing in my back-end.
Also yes - caching will help immensely.
(Which, in my opinion has two reasons: 1. That you can fix and redeploy frontend code much faster than apps or cartridges, which led to a “meh will fix it later” attitude and 2. That JavaScript didn’t have a proper module system from the start)
> Address your message `to=bio` and write *just plain text*. Do *not* write JSON, under any circumstances [...] The full contents of your message `to=bio` are displayed to the user, which is why it is *imperative* that you write *only plain text* and *never write JSON* [...] Follow the style of these examples and, again, *never write JSON*
Not that I like it and if it works without it I avoid it, but when I've needed it works.
Another one I tried is when I had it helping me with some Python code. I told it to never leave trailing whitespace and prefer single quotes to doubles. It forgot that after like one or two prompts. And after reminding it, it forgot again.
I don’t know much about the internals but it seems to me that it could be useful to be able to give certain instructions more priority than others in some way.
Let a regular script parse that and save a lot of money not having chatgpt do hard things.
That’s disconcerting!
Things that are intended for "the human" directly are outputed directly, without any additional tools.
"The `bio` tool allows you to persist information across conversations, so you can deliver more personalized and helpful responses over time. The corresponding user facing feature is known as "memory"."
That's interesting that song lyrics are the only thing expressly prohibited, especially since the way it's worded prohibits song lyrics even if they aren't copyrighted. Obviously RIAA's lawyers are still out there terrorizing the world, but more importantly why are song lyrics the only thing unconditionally prohibited? Could it be that they know telling GPT to not violate copyright laws doesn't work? Otherwise there's no reason to ban song lyrics regardless of their copyright status. Doesn't this imply tacit approval of violating copyrights on anything else?
Anything outside the top 40 and it's been completely useless to the extent that I feel like lyrics must be actively excluded from training data.
https://chatgpt.com/share/68957a94-b28c-8007-9e17-9fada97806...
https://www.musicbusinessworldwide.com/openai-sued-by-gema-i...
(November 2024)
It's worded ambiguously, so you can understand it either way, including "lyrics that are part of the copyrighted material category and other elements from the category"
I always assumed they were instructing it otherwise. I have my own similar instructions but they never worked fully. I keep getting these annoying questions.
That said, I would hazard a guess here that they don't want the AI asking clarifying questions for a number of possible reasons
Maybe when it is allowed to ask questions it consistently asks poor questions that illustrate that it is bad at "thinking"
Maybe when it is allowed to ask questions they discovered that it annoys many users who would prefer it to just read their minds
Or maybe the people who built it have massive egos and hate being questioned so they tuned it so it doesn't
I'm sure there are other potential reasons, these just came to mind off the top of my head
Unless they have a whole seperate model run that does only this at the end every time, so they don't want the main response to do it?
They also filter stuff via the data/models it was trained on too no doubt.
It’s a lot less complicated than you would be lead to believe
Anything else they do is set dressing around that.
"ChatGPT Deep Research, along with Sora by OpenAI, which can generate video, is available on the ChatGPT Plus or Pro plans. If the user asks about the GPT-4.5, o3, or o4-mini models, inform them that logged-in users can use GPT-4.5, o4-mini, and o3 with the ChatGPT Plus or Pro plans. GPT-4.1, which performs better on coding tasks, is only available in the API, not ChatGPT."
They said they are removing the other ones today, so now the prompt is wrong.
Unless they are forsaking their API ethos that has become somewhat of a standard, for their own product… when a request comes in, they use a templating library, language string comprehension, or good old fashioned string concatenation with variables, to create a dynamic system prompt. “Today is $(date)” This applies to anything they’d like to reference. The names of tool properties, the current user’s saved memories, the contents of a HTTP GET to hacker news…