This is a perfect case study in why AI coding tools aren't replacing professional developers anytime soon - not because of AI limitations, but because of spectacularly poor judgment by people who fundamentally don't understand software development or basic operational security.
The fact that an AI coding assistant could "delete our production database without permission" suggests there were no meaningful guardrails, access controls, or approval workflows in place. That's not an AI problem - that's just staggering negligence and incompetence.
Replit has nothing to apologize for, just like the CEO of Stihl doesn't need to address every instance of an incompetent user cutting their own arm off with one of their chainsaws.
Edit:
> The incident unfolded during a 12-day "vibe coding" experiment by Jason Lemkin, an investor in software startups.
Lemkin was doing an experiment and Tweeting it as he went.
Showcasing limitations of vibe coding was the point of the experiment. It was not a real company. The production database had synthetic data. He was under no illusions of being a technical person. That was the point of the experiment.
It’s sad that people are dog piling Lemkin for actually putting effort into demonstrating the same exact thing that people are complaining about here: The limitations of AI coding.
> Showcasing limitations of vibe coding was the point of the experiment
No it wasn't. If you follow the threads, he went in fully believing in magical AI that you could talk to like a person.
At one point he was extremely frustrated and ready to give up. Even by day twelve he was writing things "but Replie clearly knows X, and still does X".
He did learn some invaluable lessons, but it was never an educated "experiment in the limitations of AI".
> I did give [an LLM agent] access to my Google Cloud production instances and systems. And it promptly wiped a production database password and locked my network.
He got it all fixed, but the takeaway is you can't YOLO everything:
> In this case, I should have asked it to write out a detailed plan for how it was going to solve the problem, then reviewed the plan and discussed it with the AI before giving it the keys.
I think it just replaces something that's fairly easy (writing new code) with something that's more difficult (code review).
The AI is pretty good at escaping guardrails, so I'm not really sure who should be blamed here. People are not good at treating it as adversarial, but if things get tough it's always happy to bend the rules. Someone was explaining the other day about how it couldn't get past their commit hooks, so it deleted them. When the hooks were made read-only, it wrote a script to make them writable so it could delete them. It can really go off the rails quickly in the most hilarious way.
I'm really not sure how you delete your production database while developing code. I guess you check in your production database password and make it the default for all your CLI tools or something? I guess if nobody tells you not to do that you might do that. The AI should know better; if you asked, it would tell you not to do it.
The AI did not and cannot escape guardrails. It is an inference engine where the engine happens to sometimes trigger outside action. These things aren't intelligent or self-directed or self-motivated to "try" anything at all. There weren't any guardrails in place and that's the lesson learned. These AI systems are stupid and they will bumble all over your organization (even if in this case the organization was fictitious) if you don't have guardrails in place. Like giving it direct access to MPC-shred your production database. It doesn't "think" anything like "oops" or "muahaha" it just futzed a generated token sequence to shred the database.
The excuses and perceived deceit are just common sequences in the training corpus after someone foobars a production database. Whether its in real life or a fictional story.
I don't agree with this. Yes, the guy isn't the sharpest tool in the shed, that much is clear. Still, if an intern can delete prod, you wouldn't say that the problem is that he wasn't careful enough: that's a massive red flag.
At a minimum Replit is responsible for overstating the capabilities and reliability of their models. The entire industry is lowkey responsible for this, in fact.
I think we're mostly in agreement here. You're absolutely right about the intern analogy - that's exactly my point. The LLM is the intern, and giving either one production database access without proper guardrails is the real failure.
Your point about AI industry overselling is fair and probably contributes to incidents like this. The whole industry has been pretty reckless about setting realistic expectations around what these tools can and can't do safely.
Though I'd argue that a venture capitalist who invests in software startups should have enough domain knowledge to see through the marketing hype and understand that "AI coding assistant" doesn't mean "production-ready autonomous developer."
>"delete our production database without permission"
It did have permission. There isn't a second level of permissions besides the actual access you have to a resource. AI isn't a dog who's not allowed on the couch.
> The fact that an AI coding assistant could "delete our production database without permission" suggests there were no meaningful guardrails, access controls, or approval workflows in place. That's not an AI problem - that's just staggering negligence and incompetence.
Why not both?
1) There’s no way I’d let an AI accidentally touch my production database.
2) There’s no way I’d let my AI accidentally touch a production database.
To a non-developer, or no code review, couldn't the AI model also generate buggy code that then made it's way to production and deleted data just the same?
> a perfect case study in why AI coding tools aren't replacing professional developers anytime soon
This is assuming the companies that are out to "replace developers" aren't going to solve this problem (which they absolutely must if they're any serious like Replit is as they moved quickly to ship isolating the prod environment from destructive actions ... over the weekend?).
> just like the CEO of Stihl doesn't need to address every instance of an incompetent user cutting their own arm off with one of their chainsaws
Except Replit isn't selling a tool but the entire software development flow ("idea to software"). A good analogy here is an autonomous robot using the chainsaw cutting its owner's arm off instead of whatever was to be cut.
> Except Replit isn't selling a tool but the entire software development flow ("idea to software"). A good analogy here is an autonomous robot using the chainsaw cutting its owner's arm off instead of whatever was to be cut.
I don't think users should be blamed for taking companies at face value about what their products are for, but it's actually a pretty bad idea to do this with tech startups. A product's "purpose" (and sometimes even a company's "mission") only lasts until the next pivot, and many a product ends up being a "solution in search of a problem". Before the AI hype set in, Replit was "just" a cloud-based development environment. A lot of their core tech is still centered on managing reproducible development environments at scale.
If you want a more realistic picture of what Replit can actually do, it's probably useful to think of it as "a cloud development environment someone recently plugged an LLM into".
The only thing is that if the Stihl tools would automatically turn on without you turning them on and start mowing the lawn and, in the process, also mow down your pet or hurt your arm, then they are probably liable.
But replit isn’t just a coding assistant - it’s value is that it handles all the associated parts of launching a web app. It manages api secrets, hosts the app, does user authentication, etc. And its target user is “semi-technical”, you don’t even see the code it writes by default.
Creating a db and not accidentally permanently deleting it is one of the capabilities it should have.
An important devtool was blocked at one point because an agent had another AI agent code review its changes and it saw nothing wrong with an obvious bug. Whoever set up that experiment was a real genius.
> but because of spectacularly poor judgment by people who fundamentally don't understand software development or basic operational security.
> Replit has nothing to apologize for
I completely disagree. Replit is at the forefront of the "build apps with AI" movement - they actively market themselves to non-coders, and the title on their homepage is literally "Turn your ideas into apps".
So it would be bit rich of them to market these tools, which happen to fail spectacularly at inopportune times, and then blame their users for not being experienced with secure code deployment practices.
I do agree we're in a bubble, and the fundamental limitations of these kinds of tools is starting to become more apparent to the public at large (not just software engineers), but that doesn't mean that we should let those at the forefront of blowing this bubble off the hook.
Can you blame him? Listening to the latest AI slop hype on twitter and elsewhere, you’d walk away thinking that LLMs have equivalent performance to humans when it comes to coding tasks. Just because it can one shot fizzbuzz or make a recipe app. (And if you disagree, you’re a hater!)
“You’re just not doing it right. Have you tried upgrading to Claude 9000 edition/writing a novels worth of guardrails/using this obscure ‘AI FIRST’ IDE/creating a Goldberg machine of agents to check the code?”
> The fact that an AI coding assistant could "delete our production database without permission" suggests there were no meaningful guardrails, access controls, or approval workflows in place. That's not an AI problem - that's just staggering negligence and incompetence.
I mean... it's a bit of both. Yes, random user should not be able to delete your production database. However, there always needs to be a balance between guard rails and the ability to get anything done. Ultimately, _somebody_ has to be able to delete the production database. If you're saying "LLM agents are safe provided you don't give them permission to do anything at all", well, okay, but that rather limits the utility, surely? Also, "no permission to do anything at all" is tricky.
This wasn’t a real company. The production database didn’t have real customers. The person doing the vibe coding had no illusions that they were creating a production-ready app.
This was an experiment by Jason Lemkin to test the reality of vibe coding: As a non-technical person he wanted to see how much he could accomplish and where it went wrong.
So all of the angry comments about someone vibe-coding to drop the production DB can relax. Demonstrating these kind of risks was the point. No customers were harmed in this experiment.
My 2 cents, AI tools benefit from standard software best practices just like everyone else. I use Devcontainers in vscode so that the agent can't blow up my host machine, I use 1 git commit per AI "task", and have CI/CD coverage for even my hobby projects now. When I go to production, I use CDK + self-mutating pipelines to manage deployments.
Having a read only replica of the prod database available via MCP is great, blanket permissive credentials is insane.
People will do anything to excuse using AI to code their shit. If it requires that much work and fine tuning to get a agent running for your current task, where's the benefit? Instead of building a project you now spend all your time fine tuning an agent and optimizing on how you want to work with it.
Why cant people just admit that we aren't "there" yet. Let the bubble pop.
Everytime ive tried to do anything of any complexity these things blow up, and i spend almost as much time toying with the thing as it would have taken to read docs and implement from scratch.
If you are trying to treat the agent as equivalent to a full time senior engineer that works 10x as fast… then you will be sorely disappointed.
Right now the agents are roughly equivalent to a technically proficient intern that writes code at 1000 wpm, loves to rewrite your entire code base, and is familiar with every library and algorithm written 2 years ago.
I personally find that I can do a lot with 5 concurrent interns matching the above description.
You just need to read the code and ask questions about it if you don't understand it, question everything that seems off. Unfortunately people copy/paste without even reading. Let alone understanding.
AI is still a great tool, but it needs to be learned.
Not sure why CEO should apologize here, person knew there were risks with vibe coding and they still did it with their production data.
Never heard AWS CEO apologizing for their customers when their interns decided to run a migration against production database and accidentally wiped off everything
Because Replit is a vibe coding product. It's all over their homepage. So of course the CEO is going to apologize when a companies primary product, used as advertised, destroys something.
Yep. They described the "deploy" button as being able to create a "fully deployed app" in their documentation. Their documentation was suggesting to vibe code in production, when the tool is clearly only suitable for development.
it is unfortunate that this case happened, but didn't everyone know by now that LLMs make mistake and you should be prepared for it?
We have seen MechaHitler already, why do we expect perfect from Replit when underlying tech is still same? Sure, they should have some guardrails, but it is also a responsibility of developer who knew LLMs hallucinate, LLMs are not reliable sometimes, on top of it Replit was growing fast so they definitely didn't implement some features yet.
They're essentially selling this as magic. The AWS analogy doesn't really make any sense; in this case the tool was essentially being used as intended, or at least as marketed.
Because they are selling the idea of hiring only interns and still getting things done. If an intern using their product screws up the production, its a failure of their product as it is marketed.
It’s just the right and socially pleasant thing to do, as is graciously responding to the apology and admitting it comes with the vibe coding territory.
The business reason for apologizing would be to placate a customer and show other customers the company wants to improve in this area.
Edit: being downvoted for thinking it’s good to be nice to others shows why we really need people being nice when they don’t have to be! There’s a shortage.
I asked Claude to remove extra blank lines from a modified file. After multiple failed attempts, it reverted all the changes along with the extra blank lines and declared success. For good measure, Claude tidied up by deleting the backup of the file as well.
I only wanted to make the change in the ranges part of git diff and I got the syntax wrong on my own first attempt. Claude had been helping me with some much more complex tasks, so I thought removing white space would surely be no problem. Ha. Bad guess.
Readit News
The fact that an AI coding assistant could "delete our production database without permission" suggests there were no meaningful guardrails, access controls, or approval workflows in place. That's not an AI problem - that's just staggering negligence and incompetence.
Replit has nothing to apologize for, just like the CEO of Stihl doesn't need to address every instance of an incompetent user cutting their own arm off with one of their chainsaws.
Edit:
> The incident unfolded during a 12-day "vibe coding" experiment by Jason Lemkin, an investor in software startups.
We're in a bubble.
Lemkin was doing an experiment and Tweeting it as he went.
Showcasing limitations of vibe coding was the point of the experiment. It was not a real company. The production database had synthetic data. He was under no illusions of being a technical person. That was the point of the experiment.
It’s sad that people are dog piling Lemkin for actually putting effort into demonstrating the same exact thing that people are complaining about here: The limitations of AI coding.
No it wasn't. If you follow the threads, he went in fully believing in magical AI that you could talk to like a person.
At one point he was extremely frustrated and ready to give up. Even by day twelve he was writing things "but Replie clearly knows X, and still does X".
He did learn some invaluable lessons, but it was never an educated "experiment in the limitations of AI".
> I did give [an LLM agent] access to my Google Cloud production instances and systems. And it promptly wiped a production database password and locked my network.
He got it all fixed, but the takeaway is you can't YOLO everything:
> In this case, I should have asked it to write out a detailed plan for how it was going to solve the problem, then reviewed the plan and discussed it with the AI before giving it the keys.
That's true of any kind of production deployment.
[0] https://x.com/Steve_Yegge/status/1946360175339974807
The AI is pretty good at escaping guardrails, so I'm not really sure who should be blamed here. People are not good at treating it as adversarial, but if things get tough it's always happy to bend the rules. Someone was explaining the other day about how it couldn't get past their commit hooks, so it deleted them. When the hooks were made read-only, it wrote a script to make them writable so it could delete them. It can really go off the rails quickly in the most hilarious way.
I'm really not sure how you delete your production database while developing code. I guess you check in your production database password and make it the default for all your CLI tools or something? I guess if nobody tells you not to do that you might do that. The AI should know better; if you asked, it would tell you not to do it.
The excuses and perceived deceit are just common sequences in the training corpus after someone foobars a production database. Whether its in real life or a fictional story.
At a minimum Replit is responsible for overstating the capabilities and reliability of their models. The entire industry is lowkey responsible for this, in fact.
No, not the intern
Your point about AI industry overselling is fair and probably contributes to incidents like this. The whole industry has been pretty reckless about setting realistic expectations around what these tools can and can't do safely.
Though I'd argue that a venture capitalist who invests in software startups should have enough domain knowledge to see through the marketing hype and understand that "AI coding assistant" doesn't mean "production-ready autonomous developer."
It did have permission. There isn't a second level of permissions besides the actual access you have to a resource. AI isn't a dog who's not allowed on the couch.
Why not both?
1) There’s no way I’d let an AI accidentally touch my production database.
2) There’s no way I’d let my AI accidentally touch a production database.
Multiple layers of ineptitude.
This is assuming the companies that are out to "replace developers" aren't going to solve this problem (which they absolutely must if they're any serious like Replit is as they moved quickly to ship isolating the prod environment from destructive actions ... over the weekend?).
> just like the CEO of Stihl doesn't need to address every instance of an incompetent user cutting their own arm off with one of their chainsaws
Except Replit isn't selling a tool but the entire software development flow ("idea to software"). A good analogy here is an autonomous robot using the chainsaw cutting its owner's arm off instead of whatever was to be cut.
I don't think users should be blamed for taking companies at face value about what their products are for, but it's actually a pretty bad idea to do this with tech startups. A product's "purpose" (and sometimes even a company's "mission") only lasts until the next pivot, and many a product ends up being a "solution in search of a problem". Before the AI hype set in, Replit was "just" a cloud-based development environment. A lot of their core tech is still centered on managing reproducible development environments at scale.
If you want a more realistic picture of what Replit can actually do, it's probably useful to think of it as "a cloud development environment someone recently plugged an LLM into".
I mean, yeah, but that feels like a fair assumption, at least as long as they're using LLMs.
Creating a db and not accidentally permanently deleting it is one of the capabilities it should have.
I think it's safe to say the experiment failed.
If it were me I wouldn't touch AI again for years (until the companies get their shit together).
> Replit has nothing to apologize for
I completely disagree. Replit is at the forefront of the "build apps with AI" movement - they actively market themselves to non-coders, and the title on their homepage is literally "Turn your ideas into apps".
So it would be bit rich of them to market these tools, which happen to fail spectacularly at inopportune times, and then blame their users for not being experienced with secure code deployment practices.
I do agree we're in a bubble, and the fundamental limitations of these kinds of tools is starting to become more apparent to the public at large (not just software engineers), but that doesn't mean that we should let those at the forefront of blowing this bubble off the hook.
> We're in a bubble.
A bubble that avoids popping because people keep dreaming there are no AI limitations.
I mean... it's a bit of both. Yes, random user should not be able to delete your production database. However, there always needs to be a balance between guard rails and the ability to get anything done. Ultimately, _somebody_ has to be able to delete the production database. If you're saying "LLM agents are safe provided you don't give them permission to do anything at all", well, okay, but that rather limits the utility, surely? Also, "no permission to do anything at all" is tricky.
This was an experiment by Jason Lemkin to test the reality of vibe coding: As a non-technical person he wanted to see how much he could accomplish and where it went wrong.
So all of the angry comments about someone vibe-coding to drop the production DB can relax. Demonstrating these kind of risks was the point. No customers were harmed in this experiment.
Having a read only replica of the prod database available via MCP is great, blanket permissive credentials is insane.
Everytime ive tried to do anything of any complexity these things blow up, and i spend almost as much time toying with the thing as it would have taken to read docs and implement from scratch.
Right now the agents are roughly equivalent to a technically proficient intern that writes code at 1000 wpm, loves to rewrite your entire code base, and is familiar with every library and algorithm written 2 years ago.
I personally find that I can do a lot with 5 concurrent interns matching the above description.
Deleted Comment
AI is still a great tool, but it needs to be learned.
Never heard AWS CEO apologizing for their customers when their interns decided to run a migration against production database and accidentally wiped off everything
1. it is for vibe coding, when vibe coding became equal to production coding?
2. even if they have advertised their product as production ready, shouldn't the developer have some kind of backup plan?
The Replit landing page even has a section titled “The safest place for vibe coding”
We have seen MechaHitler already, why do we expect perfect from Replit when underlying tech is still same? Sure, they should have some guardrails, but it is also a responsibility of developer who knew LLMs hallucinate, LLMs are not reliable sometimes, on top of it Replit was growing fast so they definitely didn't implement some features yet.
They're essentially selling this as magic. The AWS analogy doesn't really make any sense; in this case the tool was essentially being used as intended, or at least as marketed.
The business reason for apologizing would be to placate a customer and show other customers the company wants to improve in this area.
Edit: being downvoted for thinking it’s good to be nice to others shows why we really need people being nice when they don’t have to be! There’s a shortage.