When I ran this experiment it was pretty exhilarating for a while. Eventually it turned into QA testing the work of a bad engineer and became exhausting. Since I had sunken so much time into it I felt pretty bad afterwards that not only did the thing it made not end up being shippable, but I hadn't benefitted as a human being while working on it. I had no new skills to show. It was just a big waste of time.
So I think the "second way" is good for demos now. It's good for getting an idea of what something can look like. However, in the future I'll be extremely careful about not letting that go on for more than a day or two.
I believe the author explicitly suggests strategies to deal with this problem, which is the entire second half of the post. There’s a big difference between when you act as a human tester in the middle vs when you build out enough guardrails that it can do meaningful autonomous work with verification.
I'm just extremely skeptical about that because I had many ideas like that and it still ended up being miserable. Maybe with Opus 4.5 things would go better though. I did choose an extremely ambitious project to be fair. If I were to try it again I would pick something more standard and a lot smaller.
+1... like with a large enough engineering team, this is ultimately a guardrails problem, which in my experience with agentic coding it’s very solvable, at least in certain domains.
This happened to me too in an experimental project where I was testing how far the model could go on its own. Despite making progress, I can't bare to look at the thing now. I don't even know what questions to ask the AI to get back into it, I'm so disconnected from it. Its exhausting to think about getting back into it; id rather just start from scratch.
The fascinating thing was how easy it was to lose control. I would set up the project with strict rules, md files and tell myself to stay fully engaged, but out of nowhere I slid into compulsive accept mode, or worse told the model to blatantly ignore my own rules I set out. I knew better, but yet it happened over and over. Ironically, it was as if my context window was so full of "successes" I forgot my own rules; I reward-hacked myself.
Maybe it just takes practice and better tooling and guardrails. And maybe this is the growing pains of a new programmers mindset. But left me a little shy to try full delegation any time soon, certainly not without a complete reset on how to approach it.
I’ll chime in to say that this happened to me as well.
My project would start good, but eventually end up in a state where nothing could be fixed and the agent would burn tokens going in circles to fix little bugs.
So I’d tell the agent to come up with a comprehensive refactoring plan that would allow the issues to be recast in more favorable terms.
I’d burn a ton of tokens to refactor, little bugs would get fixed, but it’d inevitably end up going in circles on something new.
> not only did the thing it made not end up being shippable
The difference between then and now is that often with the latest models, it is shippable without bugs within a couple of LLM reviews.
I’m ok doing the work of a dev manager while holding the position of developer.
I’m sure there was someone that once said “The washing machine didn’t do a good job, and I wasn’t proud of myself when I was using it”, but that didn’t stop washing machines from spreading to most homes in first-world countries.
I've had the opposite results. I used to "vibe code" in languages that I knew, so that I could review the code and, I assumed, contribute myself. I got good enough results that I started using AI to build tools in languages I had no prior knowledge of. I don't even look at the code any more. I'm getting incredible results. I've been a developer for 30+ years and never thought this would be possible. I keep making more and more ambitious projects and AI just keeps banging them out exactly how I envision them in my mind.
To be fair I don't think someone with less experience could get these results. I'm leveraging every thing I know about writing software, computer science, product development, team management, marketing, written communication, requirements gathering, architecture... I feel like vibe coding is pushing myself and AI to the limits, but the results are incredible.
It's also good for quickly creating legitimate looking scam and SEO spam sites. When they stop working, throw them away, and create a dozen more. Maintenance is not a concern. Scammers love this new tech.
> As AI edges humans out of the business of thinking, I think we need to be wary of losing something essential about being human
If AI edges humans out of the business of thinking, then we're all in deep shit, because it doesn't think, it just regurgitates previous human thinking. With no humans thinking, no advances in code will be possible. It will only be possible to write things which are derivatives of prior work
(cue someone arguing with me that everything humans do is a derivative of prior work)
> For 99% of tasks I'm totally certain there's people out there that are orders of magnitude better at them than me.
And LLMs slurped some of those together with the output of thousands of people who’d do the task worse, and you have no way of forcing it to be the good one every time.
> If the AI can regurgitate their thinking, my output is better.
But it can’t. Not definitively and consistently, so that hypothetical is about as meaningful as “if I had a magic wand to end world hunger, I’d use it”.
> Humans may not need to think to just... do stuff.
If you don’t think to do regular things, you won’t be able to think to do advanced things. It’s akin to any muscle; you don’t use it, it atrophies.
I was thinking about this after watching YouTube short verticals for about 2 hours last night: ~2min clips from different TV series, movies, SNL skits, music insider clips (Robert Trujillo auditions for Metallica, 2003. LOL). My friends and I often relate in regurgitated human sound bites. Which is fine when I’m sitting with friends driving to a concert. Just wasting time.
I’m thinking about this time suck, and my continual return/revisiting to my favorite hard topics in philosophy over and over. It’s certainly what we humans do. If I think deeply and critically about something, it’s from the perspective of a foundation I made for myself from reading and writing, or it was initialized by a professor and coursework.
Isn’t it all regurgitated thinking all the way down?
Creative thinking requires an intent to be creative. Yes, it may be a delusion to imagine oneself as creative, one's thoughts to be original, but you have to begin with that idea if you're going to have any chance of actually advancing human knowledge. And the stronger wider higher you build your foundation-- your knowledge and familiarity with the works of humans before your time-- the better your chance of successful creativity, true originality, immortality.
Einstein thinks nothing of import without first consuming Newton and Galileo. While standing on their shoulders, he could begin to imagine another perspective, a creative reimaging of our physical universe. I'm fairly sure that for him, like for so many others, it began as a playful, creative thought stream, a What If juxtoposition between what was known and the mystery of unexplored ideas.
Your intent to create will make you creative. Entertain yourself and your thoughts, and share when you dare, and maybe we'll know if you're regurgitating or creating. But remember that you're the first judge and gatekeeper, and the first question is always, are you creative?
> If AI edges humans out of the business of thinking, then we're all in deep shit
Also because we live under capitalism, and you need something people need you to do to be allowed to live.
For a century+, "thinking" was the task that was supposed to be left to humans, as physical labor was automated. If "AI edges humans out of the business of thinking" what's left for humans, especially those who still need to work for a living because they don't have massive piles of money.
If AI edges humans out of the business of thinking
This will never happen because the business of thinking is enjoyable and the humans whose thinking matters most will continue to be intrinsically motivated to do it.
> This will never happen because the business of thinking is enjoyable and the humans whose thinking matters most will continue to be intrinsically motivated to do it.
What world do you live in, where you get paid doing the things that are enjoyable to you, because they're enjoyable?
I became a big fan of David Bau about 10 years back, when I came across his conformal mapping page. My understanding of complex numbers changed forever.
There is an amusing parallel with his views on vibe coding. Back in the 90's and 2000's I noticed a pattern with the code developed by the huge influx of inexperienced programmers jumping on the dotcom bandwagon. The code can only be maintained by the same people who wrote it. There was no documentation, no intuition, no best practices and other wouldn't know where to fix if there is any issue. Probably the code aligned with the programmer's cultural habits and values (what's ok, what's not ok), which others might lack. Ironically, this has kind of provided job security for them, as it is difficult for others, to deal with that code.
I guess the LLMs are also into this "job-security" trick, by ensuring only LLMs can manage the LLMs generated code.
I find it's ok to vibe code something digestible like a ZSH function to do X or Y. An image converter, or something along those lines.
Anything that involves multiple days of work, or that you plan on working on it further, should absolutely not be vibe coded.
A) you'll have learnt pretty much nothing, or will retain nothing. Writing stuff by hand is a great way to remember. A painful experience worthwhile of having is one you've learnt from.
B) you'll find yourself distanced from the project and the lack personal involvement of 'being in the trenches' means you'll stop progressing on the software and move on back to something that makes you feel something.
Humans are by nature social creatures, but alone they want to feel worthwhile too. Vibe coding takes away from this positive reinforcement loop that is necessary for sticking with long running projects to achievement.
Emotions drive needs, which drives change and results. By vibe coding a significant piece of work, you'll blow away your emotions towards it and that'll be the end of it.
For 'projects' and things running where you want to be involved, you should be in charge, and only use LLMs for deterministic auto-completion, or research, outside of the IDE. Just like managing state in complex software, you need to manage LLMs' input to be 'boxed in' and not let it contaminate your work.
My 5c. Understanding the human's response to interactions with the machines is important in understanding our relationship with LLMs.
I get a huge emotional reward by conjuring up something that I dreamed of but wouldn't have had time to build otherwise. The best description I can give is back in the day when you would beat a video game to see the ending.
not to be antagonistic but are we paid to learn stuff or to build stuff? I think it's the latter. if we have to learn something it's only so that we can build something in the end.
I am absolutely paid by the hour to learn stuff. The things I'm learning are mostly messy business domain bits: how does our internal API work, who wrote it, what were the constraints, which customer requested this feature, how much SLA will we lose if we break it to hotfix this CVE...
Yes the end result is at some small moment in time a thing that was built. But the value prop of the company isn't the software, it's the ability to solve business problems. The software is a means to that end. Understanding the problems is almost the entire job.
- Is the work easier to do? I feel like the work is harder.
- Is the work faster? It sounds like it’s not faster.
- Is the resulting code more reliable? This seems plausible given the extensive testing, but it’s unclear if that testing is actually making the code more reliable than human-written code, or simply ruling out bugs an LLM makes but a human would never make.
I feel like this does not look like a viable path forward. I’m not saying LLMs can’t be used for coding, but I suspect that either they will get better, to the point that this extensive harness is unnecessary, or they will not be commonly used in this way.
I have a feeling that the real work of writing a complex application is in fully understanding the domain logic in all its gory details and creating a complete description of your domain logic in code. This process that OP is going through seems to be "what if I materialize the domain logic in tests instead of in code." Well, at first blush, it seems like maybe this is better because writing tests is "easier" than writing code. However, I imagine the biggest problem is that sometimes it takes the unyielding concreteness of code to expose the faults in your description of the domain problem. You'd end up interacting with an intermediary, using the tests as a sort of interpreter as you indirectly collaborate with the agent on defining your application. The cost of this indirection may be the price to pay for specifying your application in a simpler, abstracted form. All this being said, I would expect the answers to "is it easier? is it faster?" would be: well, it depends. If it can be better, it's certainly not always better.
Its not mutually exclusive. We write test precisely because expressing a complex application is hard without them. But to your point, we should not wave away applications that cannot be understood with extra tests. I agree.
> - Is the work faster? It sounds like it’s not faster.
The author didn't discuss the speed of the work very much. It is certainly true that LLMs can write code faster than humans, and sometimes that works well. What would be nice is an analysis of the productivity gains from LLM-assisted coding in terms of how long it took to do an entire project, start to finish.
I've been asking for little tutorials or implementation plans for things, and demanding that the model not write any code itself. Following the advice of Geoffrey Litt.[1] I find reviewing code written by my coworkers to be difficult when i'm being paid for it, surely i'm not gonna review thousands of lines of auto-generated code and the comprehensive tests required to trust them in my free time...!
So I've been learning kotlin & android development in the evenings and i find this style of thing to be so much more effective as a dev practice than claude code and a better learning practice than following dev.to tutorials. I've been coding for almost 20 years and find most tutorial or documentation stuff either targeted to someone who has hardly programmed at all, or just plain old API docs.
Asking the langlemangler to generate a dev plan, focusing on idiomatic implementation details and design questions rather than lines of code, and to let me fill in the algorithm implementations, it's been nice. I'll use the jetbrains AI autocomplete stuff for little things or ask it to refactor a stinky function but mostly I just follow the implementation plan so that the shape of the whole system is in my head.
Here's an example:
> i have scaffolded out a new project, an implementation of a library i've written multiple times in the last decade in multiple languages, but with a language i haven't written and with new design requirements specified in the documentation. i want you to write up an implementation plan, an in-depth tutorial for implementing the requirements in a Kotlin Multi Platform library.
> i am still learning kotlin but have been programming for 20 years. you don't need to baby me, but don't assume i know best practices and proper idioms for kotlin. make sure to include background context, best practices, idioms, and rationale for the design choices and separation of concerns.
This produced a 3kb markdown file that i've been following while I develop this project.
This is a really great idea going to try this out. I similarly just cannot mentally stand reviewing vibe coding PRs all day, but this sounds genuinely useful.
As someone new to this topic, I'd find it interesting to see the actual chat/cli log of a successful LLM-created project like this one, especially when it comes to the meta-layers of tests and testing the tests, etc.
Can someone recommend any good resources on this? Google wasn't too helpful, or my google-fu is lacking.
Readit News
When I ran this experiment it was pretty exhilarating for a while. Eventually it turned into QA testing the work of a bad engineer and became exhausting. Since I had sunken so much time into it I felt pretty bad afterwards that not only did the thing it made not end up being shippable, but I hadn't benefitted as a human being while working on it. I had no new skills to show. It was just a big waste of time.
So I think the "second way" is good for demos now. It's good for getting an idea of what something can look like. However, in the future I'll be extremely careful about not letting that go on for more than a day or two.
I put like 400 hours into it by the way.
The fascinating thing was how easy it was to lose control. I would set up the project with strict rules, md files and tell myself to stay fully engaged, but out of nowhere I slid into compulsive accept mode, or worse told the model to blatantly ignore my own rules I set out. I knew better, but yet it happened over and over. Ironically, it was as if my context window was so full of "successes" I forgot my own rules; I reward-hacked myself.
Maybe it just takes practice and better tooling and guardrails. And maybe this is the growing pains of a new programmers mindset. But left me a little shy to try full delegation any time soon, certainly not without a complete reset on how to approach it.
My project would start good, but eventually end up in a state where nothing could be fixed and the agent would burn tokens going in circles to fix little bugs.
So I’d tell the agent to come up with a comprehensive refactoring plan that would allow the issues to be recast in more favorable terms.
I’d burn a ton of tokens to refactor, little bugs would get fixed, but it’d inevitably end up going in circles on something new.
The difference between then and now is that often with the latest models, it is shippable without bugs within a couple of LLM reviews.
I’m ok doing the work of a dev manager while holding the position of developer.
I’m sure there was someone that once said “The washing machine didn’t do a good job, and I wasn’t proud of myself when I was using it”, but that didn’t stop washing machines from spreading to most homes in first-world countries.
To be fair I don't think someone with less experience could get these results. I'm leveraging every thing I know about writing software, computer science, product development, team management, marketing, written communication, requirements gathering, architecture... I feel like vibe coding is pushing myself and AI to the limits, but the results are incredible.
It's also good for quickly creating legitimate looking scam and SEO spam sites. When they stop working, throw them away, and create a dozen more. Maintenance is not a concern. Scammers love this new tech.
Dead Comment
If AI edges humans out of the business of thinking, then we're all in deep shit, because it doesn't think, it just regurgitates previous human thinking. With no humans thinking, no advances in code will be possible. It will only be possible to write things which are derivatives of prior work
(cue someone arguing with me that everything humans do is a derivative of prior work)
BUT. For 99% of tasks I'm totally certain there's people out there that are orders of magnitude better at them than me.
If the AI can regurgitate their thinking, my output is better.
Humans may need to think to advance the state of the art.
Humans may not need to think to just... do stuff.
And LLMs slurped some of those together with the output of thousands of people who’d do the task worse, and you have no way of forcing it to be the good one every time.
> If the AI can regurgitate their thinking, my output is better.
But it can’t. Not definitively and consistently, so that hypothetical is about as meaningful as “if I had a magic wand to end world hunger, I’d use it”.
> Humans may not need to think to just... do stuff.
If you don’t think to do regular things, you won’t be able to think to do advanced things. It’s akin to any muscle; you don’t use it, it atrophies.
And if the average person is orders of magnitude better than you at thinking, you're right... you should let the AI do it lol
Unfortunately that’s not where we’re headed.
God forbid we should ever have to think lol
I was thinking about this after watching YouTube short verticals for about 2 hours last night: ~2min clips from different TV series, movies, SNL skits, music insider clips (Robert Trujillo auditions for Metallica, 2003. LOL). My friends and I often relate in regurgitated human sound bites. Which is fine when I’m sitting with friends driving to a concert. Just wasting time.
I’m thinking about this time suck, and my continual return/revisiting to my favorite hard topics in philosophy over and over. It’s certainly what we humans do. If I think deeply and critically about something, it’s from the perspective of a foundation I made for myself from reading and writing, or it was initialized by a professor and coursework.
Isn’t it all regurgitated thinking all the way down?
Creative thinking requires an intent to be creative. Yes, it may be a delusion to imagine oneself as creative, one's thoughts to be original, but you have to begin with that idea if you're going to have any chance of actually advancing human knowledge. And the stronger wider higher you build your foundation-- your knowledge and familiarity with the works of humans before your time-- the better your chance of successful creativity, true originality, immortality.
Einstein thinks nothing of import without first consuming Newton and Galileo. While standing on their shoulders, he could begin to imagine another perspective, a creative reimaging of our physical universe. I'm fairly sure that for him, like for so many others, it began as a playful, creative thought stream, a What If juxtoposition between what was known and the mystery of unexplored ideas.
Your intent to create will make you creative. Entertain yourself and your thoughts, and share when you dare, and maybe we'll know if you're regurgitating or creating. But remember that you're the first judge and gatekeeper, and the first question is always, are you creative?
there it is
Also because we live under capitalism, and you need something people need you to do to be allowed to live.
For a century+, "thinking" was the task that was supposed to be left to humans, as physical labor was automated. If "AI edges humans out of the business of thinking" what's left for humans, especially those who still need to work for a living because they don't have massive piles of money.
What world do you live in, where you get paid doing the things that are enjoyable to you, because they're enjoyable?
What? Coding is like the one thing that RL can do without any further human input because there is a testable provable ground truth; run the code.
There is an amusing parallel with his views on vibe coding. Back in the 90's and 2000's I noticed a pattern with the code developed by the huge influx of inexperienced programmers jumping on the dotcom bandwagon. The code can only be maintained by the same people who wrote it. There was no documentation, no intuition, no best practices and other wouldn't know where to fix if there is any issue. Probably the code aligned with the programmer's cultural habits and values (what's ok, what's not ok), which others might lack. Ironically, this has kind of provided job security for them, as it is difficult for others, to deal with that code.
I guess the LLMs are also into this "job-security" trick, by ensuring only LLMs can manage the LLMs generated code.
Anything that involves multiple days of work, or that you plan on working on it further, should absolutely not be vibe coded.
A) you'll have learnt pretty much nothing, or will retain nothing. Writing stuff by hand is a great way to remember. A painful experience worthwhile of having is one you've learnt from.
B) you'll find yourself distanced from the project and the lack personal involvement of 'being in the trenches' means you'll stop progressing on the software and move on back to something that makes you feel something.
Humans are by nature social creatures, but alone they want to feel worthwhile too. Vibe coding takes away from this positive reinforcement loop that is necessary for sticking with long running projects to achievement.
Emotions drive needs, which drives change and results. By vibe coding a significant piece of work, you'll blow away your emotions towards it and that'll be the end of it.
For 'projects' and things running where you want to be involved, you should be in charge, and only use LLMs for deterministic auto-completion, or research, outside of the IDE. Just like managing state in complex software, you need to manage LLMs' input to be 'boxed in' and not let it contaminate your work.
My 5c. Understanding the human's response to interactions with the machines is important in understanding our relationship with LLMs.
Yes the end result is at some small moment in time a thing that was built. But the value prop of the company isn't the software, it's the ability to solve business problems. The software is a means to that end. Understanding the problems is almost the entire job.
Deleted Comment
- Is the work easier to do? I feel like the work is harder.
- Is the work faster? It sounds like it’s not faster.
- Is the resulting code more reliable? This seems plausible given the extensive testing, but it’s unclear if that testing is actually making the code more reliable than human-written code, or simply ruling out bugs an LLM makes but a human would never make.
I feel like this does not look like a viable path forward. I’m not saying LLMs can’t be used for coding, but I suspect that either they will get better, to the point that this extensive harness is unnecessary, or they will not be commonly used in this way.
The author didn't discuss the speed of the work very much. It is certainly true that LLMs can write code faster than humans, and sometimes that works well. What would be nice is an analysis of the productivity gains from LLM-assisted coding in terms of how long it took to do an entire project, start to finish.
So I've been learning kotlin & android development in the evenings and i find this style of thing to be so much more effective as a dev practice than claude code and a better learning practice than following dev.to tutorials. I've been coding for almost 20 years and find most tutorial or documentation stuff either targeted to someone who has hardly programmed at all, or just plain old API docs.
Asking the langlemangler to generate a dev plan, focusing on idiomatic implementation details and design questions rather than lines of code, and to let me fill in the algorithm implementations, it's been nice. I'll use the jetbrains AI autocomplete stuff for little things or ask it to refactor a stinky function but mostly I just follow the implementation plan so that the shape of the whole system is in my head.
Here's an example:
> i have scaffolded out a new project, an implementation of a library i've written multiple times in the last decade in multiple languages, but with a language i haven't written and with new design requirements specified in the documentation. i want you to write up an implementation plan, an in-depth tutorial for implementing the requirements in a Kotlin Multi Platform library. > i am still learning kotlin but have been programming for 20 years. you don't need to baby me, but don't assume i know best practices and proper idioms for kotlin. make sure to include background context, best practices, idioms, and rationale for the design choices and separation of concerns.
This produced a 3kb markdown file that i've been following while I develop this project.
[1]: https://x.com/geoffreylitt/status/1991909304085987366
Can someone recommend any good resources on this? Google wasn't too helpful, or my google-fu is lacking.
That's not vibe coding. That is just AI assisted coding.