> there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest.
It's even worse than that: non-junior devs are doing it
as well.
Yeah, it is way worse than that. In the past two days, I have had two separate non-engineer team members ask some AI agent how some mobile bug should be fixed and posted the AI response in the ticket as the main content and context and acceptance criteria. I then had to waste my time reading this crap (because this is really all that is in the ticket) before starting my own efforts to understand what the real ask or change in behavior needed is.
Our leader wrote himself a great prompt to fill up Tickets in jira with useless text too and our boss is happy like if he won the lottery. Now instead of ugly but short useful texts now i have yo read a fucking eassay!!!
You close the ticket and ping the manager of the nontechnical person submitting the ticket. Then you have a discussion with management about the arrangement and expectations. If it doesn't go well you polish your resume.
There's a difference between 10 years of experience and 1 year of experience 10 times.
YOE isn't always a measurement of quality, you can work the same dead-end coding job for 10 years and never get more than "1 year" of actual experience.
Ability is a combination of aptitude, skills, persistence, and exposure. More importantly, intention matters and it show up in the quality of your work. If the intention is to cut corners, no one can stop you from doing shoddy work. Number of years and titles do not matter much if the other parameters are low.
Aptitude paves the way for exploration: learning languages, paradigms, modeling techniques, discovering gotchas and so on. Skills follow from practice and practice requires a tough mindset where you don't give up easily.
So many software engineers learn to code just to pass exams and interviews. They claim they have strong logical reasoning. However, they have only been exposed to ideas and patterns from competitive programming rut. They have never even seen code more complex than a few hundred lines. They haven't seen problems being modeled in different languages. They haven't had the regret of building complex software without enough design. They have not had the disappointment of overengineering solutions. They have never reverse-engineered legacy code. They have never single-stepped in a debugger. All they have learned is to make random changes until "It works on my machine".
Yes, software is complex, disposable, ever-changing and often ugly but that is no excuse for keeping it that way.
This mirrors my experience with the texting while driving problem: The debate started as angry complaints about how all the kids are texting while driving. Yet it’s a common problem for people of all ages. The worst offender I knew for years was in her 50s, but even she would get angry about the youths texting while driving.
Pretending it’s just the kids and young people doing the bad thing makes the outrage easier to sell to adults.
It’s always the developers who can break / bypass the rules who are the most dangerous.
I always think of the "superstars" or "10x" devs I have met at companies. Yeah I could put out a lot of features too if I could bypass all the rules and just puke out code / greenfield code that accounts for the initial one use case ... (and sometimes even leave the rest to other folks to clean up).
Where are the junior devs while their code is being reviewed? I'm not a software developer, but I'd be loath to review someone's work unless they have enough skin in the game to be present for the review.
Code review is rarely done live. It's usually asynchronous, giving the reviewer plenty of time to read, digest, and give considered feedback on the changes.
Perhaps a spicy patch would involve some kind of meeting. Or maybe in a mentor/mentee situation where you'd want high-bandwidth communication.
A senior dev should be mentoring and talking to a junior dev about a task well before it hits the review stage. You should discuss each task with them on a high level before assigning it, so they understand the task and its requirements first, then the review is more of a formality because you were involved at each step.
As someone else mentioned, the process is async. But I achieve a similar effect by requiring my team to review their own PRs before they expect a senior developer to review them and approve for merging.
That solves some of the problem with people thinking it's okay to fire off a huge AI slop PR and make it the reviewer's responsibility to see how much the LLM hallucinated. No, you have to look at yourself first, because it's YOUR code no matter what tool you used to help write it.
>It's even worse than that: non-junior devs are doing it as well.
This might be unpopular, but that is seeming more like an opportunity if we want to continue allowing AI to generate code.
One of the annoying things engineers have to deal with is stopping whatever they're doing and doing a review. Obviously this gets worse if more total code is being produced.
We could eliminate that interruption by having someone doing more thorough code reviews, full-time. Someone who is not being bound by sprint deadlines and tempted to gloss over reviews to get back to their own work. Someone who has time to pull down the branch and actually run the code and lightly test things from an engineer's perspective so QA doesn't hit super obvious issues. They can also be the gatekeeper for code quality and PR quality.
> One of the annoying things engineers have to deal with is stopping whatever they're doing and doing a review.
I would have thought that reviewing PRs and doing it well is in the job description. You latter mention "someone" a few times - who that someone might be?
> We could eliminate that interruption by having someone doing more thorough code reviews, full-time. Someone who is not being bound by sprint deadlines and tempted to gloss over reviews to get back to their own work.
This is effectively my role (outside of mentoring) as a lead developer over a team of juniors we train in house. I'm not sure many engineers would enjoy a day of only reviewing, me included.
This sounds good in theory, but in practice, a person capable of doing a good job at this role would also be a good developer whose impact would be greater if they were churning out code. This is a combination of a lead engineer and SDET.
In reality, this ends up being the job given to the weakest person on the team to keep them occupied. And it gives the rest of the team a mechanism to get away with shoddy work and not face repercussions.
Maybe I'm just jaded, but I think this approach would have horrible results.
AI code review tools are already good. That makes for a good first pass. On my team, fixing Code Rabbit's issues, or having a good reason not to is always step 1 to a PR. It catches a lot of really subtle bugs.
>One of the annoying things engineers have to deal with is stopping whatever they're doing and doing a review
Code reviews are a part of the job. Even at the junior level, an engineer should be able to figure out a reasonable time to take a break and shift efforts for a bit to handle things like code reviews.
The best way I have of knowing code is correct on all levels is convincing myself I would write it the same way.
Thr only way to be 100% sure is writing it myself. If I know some one reasonable managed to write the code I can usually take some shortcuts and only look at the code style, common gotchas etc.
Of course it wouldn't be the first time I made some erroneous assumptions about how well considered the code was. But if none of the code is the product of any intelligent thought well, I might as well stop reading and start writing. Reading code is 10x harder than writing it after all.
It's even worse than that: as long as it somewhat works, managers don't care, if not prefer it (to more slowly developed, more tested, more well architected code)
One thing I've pushed developers on my team to do since way before AI slop became a thing was to review their own PR. Go through the PR diff and leave comments in places where it feels like a little explanation of your thought process could be helpful in the review. It's a bit like rubber duck debugging, I've seen plenty of things get caught that way.
As an upside, it helps with AI slop too. Because as I see it, what you're doing when you use an LLM is becoming a code reviewer. So you need to actually read the code and review it! If you have not reviewed it yourself first, I am not going to waste my time reviewing it for you.
It helps obviously that I'm on a small team of a half dozen developers and I'm the lead, and management hasn't even hinted at giving us stupid decrees like "now that you have Claude Code you can do 10x as many features!!!1!".
Yeah, I always think it's kinda rude to throw something to someone else to review without reviewing it yourself, even if you were the one to write it. Looking at it twice yourself can help with catching things even faster than someone else getting up to speed with what you were doing and querying it. Now it seems like with LLMs people are putting code up for review that hasn't even been looked at once.
As always, this requires nuance. Just yesterday and today, I did exactly that to my direct reports (I'm director-level). We had gotten a bug report, and the team had collectively looked into it and believed it was not our problem, but that of an external vendor. Reported it to the vendor, who looked into it, tested it, and then pushed back and said it was our problem. My team is still more LLM-averse than me, so I had Codex look at it, and it believed it found the problem and prepared the PR. I did not review or test the PR myself, but instead assigned it to the team to validate, partly for learnings. They looked it over and agreed it was a valid fix for a problem on our side. I believe that process was better than me just fully validating it myself, and part of the process toward encouraging them to use LLM as a tool for their work.
2 decades ago, so well before any LLMs, our CEO did that with a couple of huge code changes: he hacked together a few things, and threw it over the wall to us (10K lines). I was happy I did not get assigned to deal with that mess, but getting that into production quality code took more than a month!
"But I did it in a few days, how can it take so long for you guys?" was not received well by the team.
Sure, every case is its own, and maybe here it made sense if the fix was small and testing for it was simple. Personally (also in a director-level role today), I'd rather lead by example and do the full story, including testing, and especially writing automated tests (with LLM's help or not), especially if it is small (I actually did that to fix misuse of mutexes ~12 months ago in one of our platform libraries, when everybody else was stuck when our multi-threaded code behaved as single-threaded code).
Even so, I prefer to sit with them and loudly ask questions that I'd be asking myself on the path to a fix: let them learn how I get to a solution is even more valuable, IMO.
> I believe that process was better than me just fully validating it myself
Why?
> and part of the process toward encouraging them to use LLM as a tool for their work.
Did you look at it from their perspective? You set the exact opposite example and serve as a perfect example for TFA: you did not deliver code you have proven to work. I imagine some would find this demoralizing.
I've worked with a lot of director-level software folk and many would just do the work. If they're not going to do the work, then they should probably assign someone to do it.
What if it didn't work? What if you just wasted a bunch of engineering time reviewing slop? I don't comprehend this mindset. If you're supposedly a leader, then lead.
Even worse for me, some of my coworkers were doing that _before_ coding LLMs were a thing. Now LLMs are allowing them to create MRs with untested nonsense even faster which feels like a DDOS attack on my productivity.
“Move fast and break things” works well when you are a little player in a big world, because you can only perturb the system into so bad a state with you limited resources. Now, they got big, and everything is broken.
I have said it before on HN using LLMs should 100% justify devs having enough time to test and document the code, and understand it better. The problem I do see though will be management.
In the company I’m at this is beginning to happen. PM’s want to “prototype” new features and expect the engineers to finish up the work. With the expectation that it ‘just needs some polishing’. What would be your recommendation on how to handle this constructively? Flat out rejecting LLM as a prototyping tool is not an option.
This could be workable with the understanding that throwing away 100% of the prototype code is acceptable and it’s primary purpose is as a communication tool, not a technical starting point.
That startup is called CodeRabbit and damned if it doesn't come up with good suggestions sometimes. Other times you have to overrule it, or more likely create separate PRs for its suggestions, and avoid lumping a bunch of different stuff into a single PR, and sometimes it's stupid and doesn't know what it's talking about, and also misses stuff, so you do still need a human to review it. But if your at all place where LLMs are being used to generate large swaths of functional code, including tests, and human reviewers simply can't keep up, overall it does feels like a step forwards. I can't speak to how well other similar services do, but presumably they're not the only one that does that; CodeRabbit's just the one that my employer has chosen.
> expects the “code review” process to handle the rest.
The LLMs/agents have actually been doing a stellar job with code reviews. Frankly that’s one area that humans rush through, to the point it’s a running joke that the best way to get a PR granted a “lgtm” is to make it huge. I’ve almost never seen Copilot wave a PR through on the first attempt, but I usually see humans doing that.
That smells of bad team practices. Put a practical limit on PRs sizes as the first step, around 500 lines max is a good rule of thumb in my experience. Larger than that, and the expectation then is a number of small PRs to a feature branch.
I rarely see a PR that should pass without comments. Your team is being sloppy.
Quality is not rewarded at most companies, it's not going to turn into more money, it might turn into less work later, but in all likelihood, the author won't be around to reap the benefits of less work later because they will have moved onto another company.
On the contrary, since more effort doesn't yield more money, but less effort can yield the same money, the strategy is to contract the time spent on work to the smallest amount, LLMs are currently the best way to do that.
I don't see why this has to be framed as a bad thing.
Why should anyone care about the quality of software that they don't use?
If you wouldn't work on it unless you were paid to, and you can leave if and when it becomes a problem, then why spend mental energy writing even a single line?
Because nothing can beat productivity of a motivated team building code that they are proud of. The mental energy spent becomes the highest reward. As for profit, it _compounds_ as for every other business.
The fact that this is lost as a common knowledge whereas shiny examples arises regularly is very telling.
But it is not liked in business because reproducing it requires competence in the industry, and finance deep pockets don’t believe in competence anymore.
I find doing my job as best as I can intrinsically rewarding. Even tho I am getting paid peanuts and have to give more than half of those peanuts to my government. I'm that kind of sucker.
Not everything is about money. Have you never wanted to be good at something because you enjoy it? Or do something for the love of the craft? Have you heard of altruism?
For what it's worth, writing good PRs applies in more cases than just AI generated contributions. In my PR descriptions, I usually start by describing how things currently work, then a summary of what needs to change, and why. Then I go on to describe what exactly is changing with the PR. This high level summary serves to educate the reviewer, and acts as a historical record in the git log for the benefit of those who come after you.
From there, I include explicit steps for how to test, including manual testing, and unit test/E2E test commands. If it's something visual, I try to include at least a screenshot, or sometimes even a brief screen capture demonstrating the feature.
Really go out of your way to make the reviewer's life easier. One benefit of doing all of this is that in most cases, the reviewer won't need to reach out to ask simple questions. This also helps to enable more asynchronous workflows, or distributed teams in different time zones.
Also, take a moment to review your own change before asking someone else to. You can save them the trouble of finding your typos or that test logging that you meant to remove before pushing.
To be fair, copilot review is actually alright at catching these sorts of things. It remains a nice courtesy to extend to your reviewer.
I often write PR descriptions, in which I write a short explanation and try to anticipate some comments I might get. Well, every time I do, I will still get those exact comments because nobody bothers reading the description.
Not to say you shouldn't write descriptions, I will keep doing it because it's my job. But a lot of people just don't care enough or are too distracted to read them.
For many of my PR and issue comments the intended audience is myself. I find them useful even a few days later, and they become invaluable months or years later when I'm trying to understand why the code is how it is.
What is the overall practice in the team? Does everybody write good descriptions and nobody reads them, or only a few of them write good descriptions and nobody reads them?
Because if it's the latter, there's your problem: even those who write good descriptions do not expect a change request to have one, so they don't bother looking.
This is a hill I’m going to die on, but I find 9/10 times people use the pr description for what should have been comments. “Git blame” and following a link to a pr is inferior ux to source code comments.
The North Star of pr review is zero comment approvals. Comments should not be answered in line, but by pushing updates to the code. The next reader otherwise will have the exact same question and they won’t have the answer there.
The exception being comments which only make sense for the sod itself but not the new state of the code. IME that’s ~10%.
After I accepted that, I then tried to preempt the comment by just commenting myself on the function/class etc that I thought might need some elaboration...
Well, I'm sure you can guess what happened after that - within the same file even
100%. There's no difference at all in my mind between an AI-assisted PR and a regular PR: in both cases they should include proof that the change works and that the author has put the work in to test it.
At the last company I worked at (Large popular tech company) it took an act of the CTO to get engineers to simply attach a JIRA Ticket to the PR they were working on so we could track it for tax purposes.
The Devs went in kicking and screaming. As an SRE it seemed like for SDEs, writing a description of the change, explaining the problem the code is solving, testing methodology, etc is harder than actually coding. Ironically AI is proving that this theory was right all along.
You're not wrong, however the issue is that it's not always easy to detect if a PR includes proof that the change works. It requires that the reviewer interrupts what they're doing, switch context completely and look at the PR.
If you consider that reviewer bandwidth is very limited in most projects AND that the volume of low-effort-AI-assisted PR has grown incredibly over the past year, now we have a spam problem.
Some of my engineers refuse to review a patch if they detect that it's AI-assisted. They're wrong, but I understand their pain.
I do this too with our PR templates. They have the ticket/issue/story number, the description of the ask (you can copy pasta from ticket). Current state of affairs. Proposed changes. Post state of affairs. Mood gif.
I used to be much more descriptive along these lines with my PRs, but what I realized is that nobody reads the descriptions, and then drops questions that are directly answered in the description anyways.
I've found that this gets worse the longer the description is, and that a couple bullet points of the most important things gets the information across much better.
This is ~mandatory for me. Even if what I am working on is non-visual. I will take a screenshot of a new type in my IDE and put a red box around it. This conveys the focus of my attention and other important aspects of the work effort.
We should get back to the basic definition of the engineering job. An engineer understands requirements, translates them into logical flows that can be automated, communicates tradeoffs across the organization, and makes tradeoff calls on maintainability, extensibility, readability, and security. Most importantly, they’re accountable for the outcome, because many tradeoffs only reveal their cost once they hit reality
None of this is covered by code generation, nor by juniors submitting random PRs. Those are symptoms of juniors (not only) missing fundamentals.
When we forget what the job actually is, we create misalignment with junior engineers and end up with weird ideas like "spec-driven development"
If anything, coding agents are a wake-up call that clarify what engineering profession is really about
When 10K LOC AI PR's are being created, sometimes by people who either don't understand the code or haven't reviewed the code their trying to submit; the 60 million dollar failure line is potentially lying in wait.
How do you square that with "use AI and get this feature done in three days or have your 'performance reviewed' with HR in the room"? Because I'm having trouble bridging that gap.
Edit: help, the new org said the same thing. :(
Edit 2: you guys, seriously, the HR lady keeps looking up at me and shaking her head. I don't think this is good. I tried to be a real, bigboy engineer, but they just mumbled something about KPIs and put me on a PIP.
The whole reliability, etc. to many is not of much priority. Things got an absolutely shitshow and still everyone buys it.
In other words the only outcome will be that people don't have or don't want to have engineers anymore.
Companies are very much not interested in someone who does the above, but at most someone who sells or cosplays these things - if even.
Cause that what creates income. They don't care if they sell crap, they care that they sell it and the cheaper they can produce the better. So money gets poured into marketing not quality.
High quality products are not sought after. And fake quality like putting a computer or a phone in a box like jewelry, even if you throw that very box away the next time you walk by a trash bin. That's what people consider quality these days, even if it's just a waste of resources.
And businesses choose products and services the same way as regular consumers, even when they want the marketing to make them feel good about it in a slightly different way, because marketing to your target audience makes sense. Duh!
People are ready to pay more for having the premium label stamped on to something, pay more to feel good about it, but most of the time are very unwilling to pay for measurable quality, an engineer provides.
It's scary, even with infrastructure the process seems to change, probably also due to corruption, but that's a whole other can of worms.
> communicates tradeoffs across the organization
They may do that. They may be recognized for it. But if the guy next door with the right cosplay says something like "we are professionals, look at how we have been on the market for X years" or "look at our market share" then no matter how far from reality the bullshitting is they'll be getting the money.
At the beginning of the year/end of last year I learned how little expertise, professionalism and engineering are required to be a multi billion NASDAQ stock. For months I thought that it cannot possibly be, that the core product of a such a company displays such a complete lack of expertise in the core area(s). Yet, they somehow managed to convince management to just invest a couple more times of money than the original budget that was already seen as quite the stretch. Of course they promises didn't end being anywhere close to true, and they completely forgot to inform us (our management) about severe limitations.
So if you are good at selling to management which you can be by pocketing consultants recommending you then things will work seemingly no matter what.
> If anything, coding agents are a wake-up call that clarify what engineering profession is really about
I believe what we need to wake up to or come to terms with is that our industry (everything that would go into NASDAQ) is a farce. Coding agents show that. It doesn't matter to create half-assed products if you come to sell them. You are selling your products to people. Doesn't matter if it's some guy at a hot dog stand or a CEO of a big successful company or going from house to house selling the best vacuum cleaner ever. What matters is you making people believe it would be stupid not to take your product.
TBH I think Information Systems Engineering and Computer Engineering can just eat software engineers lunch at this point. the entire need for a separate engineering discipline on software was for low level coding. Custom hardware chips are easier to make for simple things and not a lot of need in low level coding anymore for more complex things means the focus is shifting back to either hardware choices or higher level system management
I'd argue the only places left you really need low level coding fall under computer science. If you are a computer or systems engineer who needs to work with a lot of code then youll benefit from having exposure to computer science, but an actual engineering discipline for just software seems silly now. Not to mention pretty much all engineers at this point are configuring software tools on their own to some degree
I think it's similar to how there used to be horse doctors as a separate profession from vets when horses were much more prominent in everyday life, but now they are all vets again and some of them specialize in horses
> I believe what we need to wake up to or come to terms with is that our industry (everything that would go into NASDAQ) is a farce.
the thing is, with software development, it's always been this way. Developers have just had tunnel vision for decades because they stare into an editor all day long instead trying to actually sell a product. If selling wasn't the top priority then what do you think would happen to your direct deposit? Software developers, especially software developers, live in this fantasy land where the believe their paycheck just happens automatically and always will. I think it's becoming critical that new software devs entering the workforce spend a couple years at a small, eat what you kill, consultancy or small business. Somewhere where they can see the relationship between building, selling, and their paycheck first hand.
I’d go further and say while testing is necessary, it is not sufficient. You have to understand the code and convince yourself that it is logically correct under all relevant circumstances, by reasoning over the code.
Testing only “proves” correctness for the specific state, environment, configuration, and inputs the code was tested with. In practice that only tests a tiny portion of possible circumstances, and omits all kinds of edge and non-edge cases.
I find myself not really trusting just tests, I really need to try the app/new function in multiple ways with the goal of breaking it. In that process I may not break it but I will notice something that might break, so I rewrite it better
If you don't push your system to failure, you can't really say you understand it. And anyways the precise failure modes under various conditions are important characteristics for stability/resiliency. (Does it shed load all the way upto network bandwidth of SYNs; allocate all the memory and then exit; freeze up with deadlocks/disk contention; go unresponsive for a few minutes then recover if the traffic dies off; answer health check pings only and not progress on actual work).
If you write your tests the Test-Driven Development way in that they first fail before production changes are introduced, you will be able to trust them a lot more. Especially if they are well-written tests that test behavior or contracts, not implementation details. I find that dependency injection helps a lot with this. I try to avoid mocking and complex dependencies as much as possible. This also allows me to easily refactor the code without having to worry about breaking anything if all the tests still pass.
When it comes to agentic coding. I created an open source tool that enforces those practices. The agent gets blocked by a hook if it tries to do anything that violates those principles. I think it helps a lot if I may say so myself.
Since we can't really formally prove most code, I think property based testing such as with hypothesis[1] would make sense.
I have not used it yet, but am about to for stuff that really needs to work.
It helps some. There are plenty of errors, a large majority I'd say, where types don't help at all. Types don't free up memory or avoid off-by-one errors or keep you from mixing up two counter variables.
If your acceptance criteria state something like “produces output f(x) for any inout x, where f(x) is defined as follows: […]”, then you can’t possibly test that, because you can’t test all possible values of x. And if the criteria don’t state that, then they don’t cover the full specification of how the software is expected to behave, hence you have to go beyond those criteria to ensure that the software always behaves as expected.
You can’t prove that something is correct by example. Examples can only disprove correctness. And tests are always only examples.
Testing is not perfect, but what else is there? Even formal proofs are just another expression of testing. With greater mathematical guarantees than other expressions, granted, but still testing all the same; prone to all the very same human problems testing is burdened with.
The difference with proofs (whether formal or informal) is that they quantify over all possible cases, whereas testing is always limited to specific cases.
I'd go further and say vibe coding it up, testing the green case, and deploying it straight into the testing environment is good enough.
The rest we can figure out during testing, or maybe you even have users willing to beta-test for you.
This way, while you're still on the understanding part and reasoning over the code, your competitor already shipped ten features, most of them working.
Ok, that was a provocative scenario. Still, nowadays I am not sure you even have to understand the code anymore. Maybe having a reasonable belief that it does work will be sufficient in some circumstances.
This approach sounds like a great way to get a lot of security holes into your code.
Maybe your competitors will be faster at first, but it’s probably better to be a bit slower and not leaking all your users data.
I would like to challenge this claim. I think LLMs are maybe accurate enough that we don't need to check every line and remember everything. High level design is enough.
I've been tasked with doing a very superficial review of a codebase produced by an adult who purports to have decades of database/backend experience with the assistance of a well-known agent.
While skimming tests for the python backend, I spotted the following:
@patch.dict(os.environ, {"ENVIRONMENT": "production"})
def test_settings_environment_from_env(self) -> None:
"""Test environment setting from env var."""
from importlib import reload
import app.config
reload(app.config)
# Settings should use env var
assert os.environ.get("ENVIRONMENT") == "production"
This isn't an outlier. There are smells everywhere.
I have plenty of anecdata that counters your anecdata.
LLMs can generate code that works. That much is true. You can generate sufficiently complex projects that simply run on the first (or second try). You can even get the LLM to write tests for the code. You can prompt it for 100% test coverage and it will provide you exactly what you want.
But that doesn't mean OP isn't correct. First, you shouldnt be remembering everything. If you are finding yourself remembering everything your project is either small (I'd guess less than 1000 lines) or you are overburdened and need help. Reasoning, logically, through code you write can be done JIT as you're writing the code. LLMs even suffer from the same problem. Instead of calling it "having to remember to much" we refer to it as a quantity called "context window". The only problem is the LLM won't prompt you telling you that it's context window is so full it can't do it's job properly. A human will.
I think an engineer should always be reasoning about their code. They should be especially suspicious of LLM generated code. Maybe I'm alone but if I use an LLM to generate code I will review it and typically end up modifying it. I find even prompting with something like "the code you write should be maintainable by other engineers" doesn't produce good value.
there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest.
Is anyone else seeing this in their orgs? I'm not...
I am currently going through this with someone in our organization.
Unfortunately, this person is vibe coding completely, and even the PR process is painful:
* The coding agent reverts previously applied feedback
* Coding agent not following standards throughout the code base
* Coding agent re-inventing solutions that already exist
* PR feedback is being responded to with agent output
* 50k line PRs that required a 10-20 line change
* Lack of testing (though there are some automated tests, but their validations are slim/lacking)
* Bad error handling/flow handling
I'm seeing a little bit of this. However, I will add that the primary culprits are engineers that were submitting low quality PRs before they had access to LLMs, they can just submit them faster now.
What's the ratio of people who things the right way vs not? I mean, is it a matter of giving them feedback to remind them what a "quality PR" is? Does that help?
In that first one, the double pasted AI retort in the last comment is pretty wild. In both of these, look at the actual "files changed" tab for the wtf.
I recently reviewed a PR that I suspect is AI generated. It added a function that doesn't appear to be called from anywhere.
It's shit because AI is absolutely not on the level of a good developer yet. So it changes the expectation. If a PR is not AI generated then there is a reasonable expectation that a vaguely competent human has actually thought about it. If it's AI generated then the expectation is that they didn't really think about it at all and are just hoping the AI got it right (which it very often doesn't). It's rude because you're essentially pawning off work that the author should have done to the reviewer.
Obviously not everyone dumps raw AI generated code straight into a PR, so I don't have any problem with using AI in general. But if I can tell that your code is AI generated (as you easily can in the cases you linked), then you've definitely done it wrong.
I feel like this becomes kind of unacceptable as soon as you take on your first developer employee. 10K LOC changes from the CTO is fine when it's only the CTO working on the project.
Hell, for my hobby projects, I try to keep individual commits under 50-100 lines of code.
Lol I worked at a startup where the CTO did this. The problem was that it was pure spaghetti code. It was so bad it kept me up at night, thinking about how to fix things. I left within 30 days
I worked with a “CTO” who did that before LLMs - one of the worst jobs I have had in the last 10 years. I spent at least 50% of my time putting out fires or refactoring his garbage code
A friend of mine has a junior engineer who does this and then responds to questions like "Why did you do X?" with "I didn't, Claude did, I don't know why".
Some other comments suggest immediately firing.. but a junior engineer needs to be mentored. It should be explained to them clearly that they need to understand the changes they have made. They should also be pointed towards the coding standards and SDLC documentation. If they refuse to change their ways, then firing makes sense.
Yep. Remember, people not posting on this website are just grinding away at jobs where their individual output does not matter, and entire motivation is work JUST hard enough not to get fired. They don't get stock grants, extremely favorable stock options or anything else, they get salary and MAYBE a small bonus based off business factors they have little control over.
My eyes were wide open when 2 jobs ago, they said they would be blocking all personal web browsing from work computers. Multiple Software Devs were unhappy because they were using their work laptop for booking flights, dealing with their kids schools stuff and other personal things. They did not have personal computer at all.
I don't see most PRs because they happen in other teams, but I am part of Slack channel where there are too many "oops" messages for my liking.
I.e. 1-2 times a month, there's an SQL script posted that will be run against prod to "hopefully fix data for all customers who were put into a bad state from a previous code release".
The person who posts this type of message most often is also the one running internal demos of the latest AI flows and trying to get everyone else onboard.
I thought we were not, but we had just been lucky. A sequence of events lately have shown that the struggle is real. This was not a junior developer though, but an experienced one. Experience does not equal skill, evidently.
Definitely seeing a bit of this, but it isn't constrained to junior devs. It's also pretty solvable by explaining to the person why it's not great, and just updating team norms.
I'm seeing it on some open source projects I maintain. Recently had 10 or so PRs come in. All very valid features - but from looking at them, not actually tested.
Not at all. Submitting untested PRs is a wildly outside of my experience. Having tests written to cover your code is a pre-requisite for having your PR reviewed on our team. "Does it work" aka passing manual testing, is literally the bare minimum before submitting a PR
It isn't only junior engineers, but otherwise. It is a small number of people from all levels.
People do what they think they will be rewarded for. When you think your job is to write a lot of code then LLMs are great. When you need quality code you start to ask if LLMs are better or not?
I started seeing it from a particularly poor developer sometime last year. I was the only reviewer for him so I saw all of his PRs. He refused to stop despite my polite and then not so polite admonishments, and was soon fired for it.
But LLMs don't really perform well enough on our codebase to allow you to generate things that even appear to work. And I'm the most junior member of my team at 37 years of age, hired in 2019.
I really tried to follow the mandate from on high to use Copilot, but the Agent mode can't even write code that compiles with the tools available to it.
Luckily I hooked it up to gptel so I can at least ask it quick questions about big functions I don't want to read in emacs.
Yes, in the only successful OSS project that I “maintain.”
Fully vibe coded, which at least they admitted. And when I pointed out the thing is off by an order of magnitude, and as such doesn't implement said feature — at all — we get pressed on our AI policy, so as to not waste their time.
I don't have an AI policy, like I don't have an IDE policy, but things get ridiculous fast with vibe coding.
Similar, at my last job. And the pushback was greater because super duper clever AI helped write it, who obviously knows more than any other senior engineer could know, so they were expecting an immediate PR approval and got all uppity when you tried to suggest changes.
Yeah, a slice of this is technology related, but it's really a policy issue. It's probably easier to manage with a tighter team. Maybe I'm taking team size for granted.
Always the same old tiring "this has always been possible before in some remotely similar fashion hence we should not criticise anything ever again" argument.
You could intuitively think it's just a difference of degree, but it's more akin to a difference of kind. Same for a nuke vs a spear, both are weapons, no one argues they're similar enough that we can treat them the same way
I would bet in most organizations you can find a copy-pasted version of the top SO answer for email regex in their language of choice, and if you chase down the original commit author they couldn't explain how it works.
Yeah, but being able to produce nuclear-sized 10k+ LOC PRs to open-source projects in minutes with relatively-zero effort definitely is. At least you had to use your brain to know which blog posts/SO answers to copypasta from.
I used to do that in simpler days. I'd add a link to where I copied it from so we could reference it if there were problems. This was for relatively small projects with just a few people.
> Don’t be tempted to skip the manual test because you think the automated test has you covered already! Almost every time I’ve done this myself I’ve quickly regretted it.
Seriously, this cannot be emphasized enough. Before LLMS when we were writing tests completely manually, manual testing made sense to me as the second step. However after playing around a lot with coding agents and LLMs, I fully agree this has flipped. Test it manually first! When you generate the tests it is extremely wise to ensure that the tests fail without the new code, and pass with it. You definitely need to review the test though, because it's remarkably easy to have the agent put something in there that makes it not a good test.
Just a couple days ago for example, Claude made a test pass by skipping authentication and leaving a brief comment informing that the authentication made the test flaky. It even threw a quick variable in there that enabled running or disabling flaky tests, and flaky tests were disabled by default! Had I not been doing a good review, I definitely would have missed it because it was cleverly subtle. I've also seen it test the wrong endpoint!
Actually it's more specific than that. A company pays you not just to "write code", not just to "write code that works", but to write code that works in the real world. Not on your laptop. Not in CI tests. Not on some staging environment. But in the real world. It may work fine in a theoretical environment, but deflate like a popped balloon in production. This code has no value to the business; they don't pay you to ship popped balloons.
Therefore you must verify it works as intended in the real world. This means not shipping code and hoping for the best, but checking that it actually does the right thing in production. And on top of that, you have to verify that it hasn't caused a regression in something else in production.
You could try to do that with tests, but tests aren't always feasible. Therefore it's important to design fail-safes into your code that ALERT YOU to unexpected or erroneous conditions. It needs to do more than just log an error to some logging system you never check - you must actually be notified of it, and you should consider it a flaw in your work, like a defective pair of Nikes on an assembly line. Some kind of plumbing must exist to take these error logs (or metrics, traces, whatever) and send it to you. Otherwise you end up producing a defective product, but never know it, because there's nothing in place to tell you its flaws.
Every single day I run into somebody's broken webapp or mobile app. Not only do the authors have no idea (either because they aren't notified of the errors, or don't care about them), there is no way for me to even e-mail the devs to tell them. I try to go through customer support, a chat agent, anything, and even they don't have a way to send in bug reports. They've insulated themselves from the knowledge of their own failures.
It gets interesting when a company assigns 2 story points to a task that requires 6 minimum. No time for writing tests, barely any time to perform code reviews and QA. Also, next year the company tells you since we have AI now, all tickets must be done 2 times quicker.
Who popped this balloon? I know I need to change my employer, but it's not so easy. And I'm not sure another employer is going to be any better.
Classic butchering of otherwise decent Scrum idea. If assigning 2 points means no tests, then you are already using story points wrong, and complaining about it is meaningless.
I think the problem is in what “proven” means. People that don’t know any better will just do that all with LLMs and still deliver the giant untested PRs but with some LLM written tests attached.
I vibe code a lot of stuff for myself, mostly for viewing data, when I don’t really need to care how it works. I’m coming around to the idea that outside of some specific circumstances where everyone has agreed they don’t need to care about or understand the code, team vibe coding is a bad practice.
If I’m paying an engineer, it’s for their work, unless explicitly agreed otherwise.
I think vibe coding is soon going to be seen the same way as “research” where you engage an offshore team (common e.g. in consulting) to give you a rundown on some topic and get back the first five google search results. Everyone knows how to do that, if it’s what they wanted they wouldn’t be hiring someone to do it.
That's why I emphasized the manual testing component as well. Attaching a screenshot or video of a feature working to your PR is a great way to prove that you've actually seen it work correctly - at least once, which is still a huge improvement over it not actually working at all.
If someone opened a PR, and it obviously doesn’t work but they claim they tested it, maybe that’s ok for the first time.
The second time it happens they gotta go.
I would find the expectation that I need to attach a screenshot insulting. And the understanding that my peers test their code to produce a screenshot would be pretty demoralizing.
Yes! This is something that I also value. Having demo gifs of before and after helps a lot. I have encountered situations where what I thought was a minor finishing clean up had an effect that I didn't anticipate. By including demos in the PR it becomes a kind of guardrail against those situations for me. I also think it is neat and generally helpful for everyone.
It's even worse than that: non-junior devs are doing it as well.
There's a difference between 10 years of experience and 1 year of experience 10 times.
YOE isn't always a measurement of quality, you can work the same dead-end coding job for 10 years and never get more than "1 year" of actual experience.
My career trajectory is wild. At this rate I'll be CTO soon, then back to mid-level.
Aptitude paves the way for exploration: learning languages, paradigms, modeling techniques, discovering gotchas and so on. Skills follow from practice and practice requires a tough mindset where you don't give up easily.
So many software engineers learn to code just to pass exams and interviews. They claim they have strong logical reasoning. However, they have only been exposed to ideas and patterns from competitive programming rut. They have never even seen code more complex than a few hundred lines. They haven't seen problems being modeled in different languages. They haven't had the regret of building complex software without enough design. They have not had the disappointment of overengineering solutions. They have never reverse-engineered legacy code. They have never single-stepped in a debugger. All they have learned is to make random changes until "It works on my machine".
Yes, software is complex, disposable, ever-changing and often ugly but that is no excuse for keeping it that way.
Pretending it’s just the kids and young people doing the bad thing makes the outrage easier to sell to adults.
Deleted Comment
Dead Comment
I always think of the "superstars" or "10x" devs I have met at companies. Yeah I could put out a lot of features too if I could bypass all the rules and just puke out code / greenfield code that accounts for the initial one use case ... (and sometimes even leave the rest to other folks to clean up).
Dead Comment
Perhaps a spicy patch would involve some kind of meeting. Or maybe in a mentor/mentee situation where you'd want high-bandwidth communication.
That solves some of the problem with people thinking it's okay to fire off a huge AI slop PR and make it the reviewer's responsibility to see how much the LLM hallucinated. No, you have to look at yourself first, because it's YOUR code no matter what tool you used to help write it.
This might be unpopular, but that is seeming more like an opportunity if we want to continue allowing AI to generate code.
One of the annoying things engineers have to deal with is stopping whatever they're doing and doing a review. Obviously this gets worse if more total code is being produced.
We could eliminate that interruption by having someone doing more thorough code reviews, full-time. Someone who is not being bound by sprint deadlines and tempted to gloss over reviews to get back to their own work. Someone who has time to pull down the branch and actually run the code and lightly test things from an engineer's perspective so QA doesn't hit super obvious issues. They can also be the gatekeeper for code quality and PR quality.
This is not the first time somebody had that idea.
I would have thought that reviewing PRs and doing it well is in the job description. You latter mention "someone" a few times - who that someone might be?
This is effectively my role (outside of mentoring) as a lead developer over a team of juniors we train in house. I'm not sure many engineers would enjoy a day of only reviewing, me included.
In reality, this ends up being the job given to the weakest person on the team to keep them occupied. And it gives the rest of the team a mechanism to get away with shoddy work and not face repercussions.
Maybe I'm just jaded, but I think this approach would have horrible results.
AI code review tools are already good. That makes for a good first pass. On my team, fixing Code Rabbit's issues, or having a good reason not to is always step 1 to a PR. It catches a lot of really subtle bugs.
Code reviews are a part of the job. Even at the junior level, an engineer should be able to figure out a reasonable time to take a break and shift efforts for a bit to handle things like code reviews.
Thr only way to be 100% sure is writing it myself. If I know some one reasonable managed to write the code I can usually take some shortcuts and only look at the code style, common gotchas etc.
Of course it wouldn't be the first time I made some erroneous assumptions about how well considered the code was. But if none of the code is the product of any intelligent thought well, I might as well stop reading and start writing. Reading code is 10x harder than writing it after all.
Dead Comment
As an upside, it helps with AI slop too. Because as I see it, what you're doing when you use an LLM is becoming a code reviewer. So you need to actually read the code and review it! If you have not reviewed it yourself first, I am not going to waste my time reviewing it for you.
It helps obviously that I'm on a small team of a half dozen developers and I'm the lead, and management hasn't even hinted at giving us stupid decrees like "now that you have Claude Code you can do 10x as many features!!!1!".
"But I did it in a few days, how can it take so long for you guys?" was not received well by the team.
Sure, every case is its own, and maybe here it made sense if the fix was small and testing for it was simple. Personally (also in a director-level role today), I'd rather lead by example and do the full story, including testing, and especially writing automated tests (with LLM's help or not), especially if it is small (I actually did that to fix misuse of mutexes ~12 months ago in one of our platform libraries, when everybody else was stuck when our multi-threaded code behaved as single-threaded code).
Even so, I prefer to sit with them and loudly ask questions that I'd be asking myself on the path to a fix: let them learn how I get to a solution is even more valuable, IMO.
Why?
> and part of the process toward encouraging them to use LLM as a tool for their work.
Did you look at it from their perspective? You set the exact opposite example and serve as a perfect example for TFA: you did not deliver code you have proven to work. I imagine some would find this demoralizing.
I've worked with a lot of director-level software folk and many would just do the work. If they're not going to do the work, then they should probably assign someone to do it.
What if it didn't work? What if you just wasted a bunch of engineering time reviewing slop? I don't comprehend this mindset. If you're supposedly a leader, then lead.
Title only loosely tracks skill level and with AI, that may become even more true.
Its when your PEERS do it that its a huge problem.
Then innocently say "LLM believes this is bad architecture and should be recreated from scratch."
If i was CTO I would not be happy to hear my engineers are spending lots of time re-writing and testing code written by product managers. Big nope.
Dead Comment
[Edit] man, people dont get /s unless its explicit
Deleted Comment
I don’t know about you, but I get paychecks twice a month for doing things included in my job description.
The LLMs/agents have actually been doing a stellar job with code reviews. Frankly that’s one area that humans rush through, to the point it’s a running joke that the best way to get a PR granted a “lgtm” is to make it huge. I’ve almost never seen Copilot wave a PR through on the first attempt, but I usually see humans doing that.
I rarely see a PR that should pass without comments. Your team is being sloppy.
On the contrary, since more effort doesn't yield more money, but less effort can yield the same money, the strategy is to contract the time spent on work to the smallest amount, LLMs are currently the best way to do that.
I don't see why this has to be framed as a bad thing. Why should anyone care about the quality of software that they don't use? If you wouldn't work on it unless you were paid to, and you can leave if and when it becomes a problem, then why spend mental energy writing even a single line?
The fact that this is lost as a common knowledge whereas shiny examples arises regularly is very telling.
But it is not liked in business because reproducing it requires competence in the industry, and finance deep pockets don’t believe in competence anymore.
From there, I include explicit steps for how to test, including manual testing, and unit test/E2E test commands. If it's something visual, I try to include at least a screenshot, or sometimes even a brief screen capture demonstrating the feature.
Really go out of your way to make the reviewer's life easier. One benefit of doing all of this is that in most cases, the reviewer won't need to reach out to ask simple questions. This also helps to enable more asynchronous workflows, or distributed teams in different time zones.
To be fair, copilot review is actually alright at catching these sorts of things. It remains a nice courtesy to extend to your reviewer.
I’ll put up a draft early and use it as a place to write and refine the PR details as I wrap up, make adjustments, add a few more tests, etc.
Not to say you shouldn't write descriptions, I will keep doing it because it's my job. But a lot of people just don't care enough or are too distracted to read them.
Because if it's the latter, there's your problem: even those who write good descriptions do not expect a change request to have one, so they don't bother looking.
The North Star of pr review is zero comment approvals. Comments should not be answered in line, but by pushing updates to the code. The next reader otherwise will have the exact same question and they won’t have the answer there.
The exception being comments which only make sense for the sod itself but not the new state of the code. IME that’s ~10%.
I have bought my tombstone.
Well, I'm sure you can guess what happened after that - within the same file even
I've also never seen anybody but myself write substantial PR descriptions at my previous 4-5 jobs
The Devs went in kicking and screaming. As an SRE it seemed like for SDEs, writing a description of the change, explaining the problem the code is solving, testing methodology, etc is harder than actually coding. Ironically AI is proving that this theory was right all along.
If you consider that reviewer bandwidth is very limited in most projects AND that the volume of low-effort-AI-assisted PR has grown incredibly over the past year, now we have a spam problem.
Some of my engineers refuse to review a patch if they detect that it's AI-assisted. They're wrong, but I understand their pain.
I've found that this gets worse the longer the description is, and that a couple bullet points of the most important things gets the information across much better.
This is ~mandatory for me. Even if what I am working on is non-visual. I will take a screenshot of a new type in my IDE and put a red box around it. This conveys the focus of my attention and other important aspects of the work effort.
Deleted Comment
Oh, there are, for years :D This has really stood the test of time:
https://rfc.zeromq.org/spec/42/#24-development-process
And its rationale is well explained too:
https://hintjens.gitbooks.io/social-architecture/content/cha...
Saddened by realizing that Pieter would have had amazing things to say about AI.
None of this is covered by code generation, nor by juniors submitting random PRs. Those are symptoms of juniors (not only) missing fundamentals. When we forget what the job actually is, we create misalignment with junior engineers and end up with weird ideas like "spec-driven development"
If anything, coding agents are a wake-up call that clarify what engineering profession is really about
https://read.engineerscodex.com/p/how-one-line-of-code-cause...
When 10K LOC AI PR's are being created, sometimes by people who either don't understand the code or haven't reviewed the code their trying to submit; the 60 million dollar failure line is potentially lying in wait.
Edit: help, the new org said the same thing. :(
Edit 2: you guys, seriously, the HR lady keeps looking up at me and shaking her head. I don't think this is good. I tried to be a real, bigboy engineer, but they just mumbled something about KPIs and put me on a PIP.
The whole reliability, etc. to many is not of much priority. Things got an absolutely shitshow and still everyone buys it.
In other words the only outcome will be that people don't have or don't want to have engineers anymore.
Companies are very much not interested in someone who does the above, but at most someone who sells or cosplays these things - if even.
Cause that what creates income. They don't care if they sell crap, they care that they sell it and the cheaper they can produce the better. So money gets poured into marketing not quality.
High quality products are not sought after. And fake quality like putting a computer or a phone in a box like jewelry, even if you throw that very box away the next time you walk by a trash bin. That's what people consider quality these days, even if it's just a waste of resources.
And businesses choose products and services the same way as regular consumers, even when they want the marketing to make them feel good about it in a slightly different way, because marketing to your target audience makes sense. Duh!
People are ready to pay more for having the premium label stamped on to something, pay more to feel good about it, but most of the time are very unwilling to pay for measurable quality, an engineer provides.
It's scary, even with infrastructure the process seems to change, probably also due to corruption, but that's a whole other can of worms.
> communicates tradeoffs across the organization
They may do that. They may be recognized for it. But if the guy next door with the right cosplay says something like "we are professionals, look at how we have been on the market for X years" or "look at our market share" then no matter how far from reality the bullshitting is they'll be getting the money.
At the beginning of the year/end of last year I learned how little expertise, professionalism and engineering are required to be a multi billion NASDAQ stock. For months I thought that it cannot possibly be, that the core product of a such a company displays such a complete lack of expertise in the core area(s). Yet, they somehow managed to convince management to just invest a couple more times of money than the original budget that was already seen as quite the stretch. Of course they promises didn't end being anywhere close to true, and they completely forgot to inform us (our management) about severe limitations.
So if you are good at selling to management which you can be by pocketing consultants recommending you then things will work seemingly no matter what.
> If anything, coding agents are a wake-up call that clarify what engineering profession is really about
I believe what we need to wake up to or come to terms with is that our industry (everything that would go into NASDAQ) is a farce. Coding agents show that. It doesn't matter to create half-assed products if you come to sell them. You are selling your products to people. Doesn't matter if it's some guy at a hot dog stand or a CEO of a big successful company or going from house to house selling the best vacuum cleaner ever. What matters is you making people believe it would be stupid not to take your product.
I'd argue the only places left you really need low level coding fall under computer science. If you are a computer or systems engineer who needs to work with a lot of code then youll benefit from having exposure to computer science, but an actual engineering discipline for just software seems silly now. Not to mention pretty much all engineers at this point are configuring software tools on their own to some degree
I think it's similar to how there used to be horse doctors as a separate profession from vets when horses were much more prominent in everyday life, but now they are all vets again and some of them specialize in horses
the thing is, with software development, it's always been this way. Developers have just had tunnel vision for decades because they stare into an editor all day long instead trying to actually sell a product. If selling wasn't the top priority then what do you think would happen to your direct deposit? Software developers, especially software developers, live in this fantasy land where the believe their paycheck just happens automatically and always will. I think it's becoming critical that new software devs entering the workforce spend a couple years at a small, eat what you kill, consultancy or small business. Somewhere where they can see the relationship between building, selling, and their paycheck first hand.
Testing only “proves” correctness for the specific state, environment, configuration, and inputs the code was tested with. In practice that only tests a tiny portion of possible circumstances, and omits all kinds of edge and non-edge cases.
I like using the word "demonstrates" in almost every case where people currently use the word "proves".
A test is a demonstration of the code working in a specific case. It is a piece of evidence, but not a general proof.
And these kinds of narrow ad-hoc proofs are fine! Usually adequate.
To rephrase the title of TFA, we must deliver code that is demonstrated to work.
When it comes to agentic coding. I created an open source tool that enforces those practices. The agent gets blocked by a hook if it tries to do anything that violates those principles. I think it helps a lot if I may say so myself.
https://github.com/nizos/tdd-guard
Edit: I realize now that I misunderstood your comment. I was quick to respond.
[1] https://news.ycombinator.com/item?id=45818562
You can’t prove that something is correct by example. Examples can only disprove correctness. And tests are always only examples.
The rest we can figure out during testing, or maybe you even have users willing to beta-test for you.
This way, while you're still on the understanding part and reasoning over the code, your competitor already shipped ten features, most of them working.
Ok, that was a provocative scenario. Still, nowadays I am not sure you even have to understand the code anymore. Maybe having a reasonable belief that it does work will be sufficient in some circumstances.
How are we supposed to use software in healthcare, defense, transportation if that's the bar?
While skimming tests for the python backend, I spotted the following:
This isn't an outlier. There are smells everywhere.LLMs can generate code that works. That much is true. You can generate sufficiently complex projects that simply run on the first (or second try). You can even get the LLM to write tests for the code. You can prompt it for 100% test coverage and it will provide you exactly what you want.
But that doesn't mean OP isn't correct. First, you shouldnt be remembering everything. If you are finding yourself remembering everything your project is either small (I'd guess less than 1000 lines) or you are overburdened and need help. Reasoning, logically, through code you write can be done JIT as you're writing the code. LLMs even suffer from the same problem. Instead of calling it "having to remember to much" we refer to it as a quantity called "context window". The only problem is the LLM won't prompt you telling you that it's context window is so full it can't do it's job properly. A human will.
I think an engineer should always be reasoning about their code. They should be especially suspicious of LLM generated code. Maybe I'm alone but if I use an LLM to generate code I will review it and typically end up modifying it. I find even prompting with something like "the code you write should be maintainable by other engineers" doesn't produce good value.
Is anyone else seeing this in their orgs? I'm not...
Unfortunately, this person is vibe coding completely, and even the PR process is painful: * The coding agent reverts previously applied feedback * Coding agent not following standards throughout the code base * Coding agent re-inventing solutions that already exist * PR feedback is being responded to with agent output * 50k line PRs that required a 10-20 line change * Lack of testing (though there are some automated tests, but their validations are slim/lacking) * Bad error handling/flow handling
This is hilarious. Not when you're the reviewer, of course, but as a bystander, this is expert-level enterprise-grade trolling.
https://github.com/WireGuard/wireguard-android/pull/82https://github.com/WireGuard/wireguard-android/pull/80
In that first one, the double pasted AI retort in the last comment is pretty wild. In both of these, look at the actual "files changed" tab for the wtf.
I’d love to hear your thoughts on LLMs, Jason. How do you use them in your projects? Do they play a role in your workflow at all?
Deleted Comment
I recently reviewed a PR that I suspect is AI generated. It added a function that doesn't appear to be called from anywhere.
It's shit because AI is absolutely not on the level of a good developer yet. So it changes the expectation. If a PR is not AI generated then there is a reasonable expectation that a vaguely competent human has actually thought about it. If it's AI generated then the expectation is that they didn't really think about it at all and are just hoping the AI got it right (which it very often doesn't). It's rude because you're essentially pawning off work that the author should have done to the reviewer.
Obviously not everyone dumps raw AI generated code straight into a PR, so I don't have any problem with using AI in general. But if I can tell that your code is AI generated (as you easily can in the cases you linked), then you've definitely done it wrong.
Probs fine when you are still in the exploration phase of a startup, scary once you get to some kind of stability
Hell, for my hobby projects, I try to keep individual commits under 50-100 lines of code.
And if it was repeated... Well I would probably get fired...
And not just from juniors
My eyes were wide open when 2 jobs ago, they said they would be blocking all personal web browsing from work computers. Multiple Software Devs were unhappy because they were using their work laptop for booking flights, dealing with their kids schools stuff and other personal things. They did not have personal computer at all.
I.e. 1-2 times a month, there's an SQL script posted that will be run against prod to "hopefully fix data for all customers who were put into a bad state from a previous code release".
The person who posts this type of message most often is also the one running internal demos of the latest AI flows and trying to get everyone else onboard.
People do what they think they will be rewarded for. When you think your job is to write a lot of code then LLMs are great. When you need quality code you start to ask if LLMs are better or not?
But LLMs don't really perform well enough on our codebase to allow you to generate things that even appear to work. And I'm the most junior member of my team at 37 years of age, hired in 2019.
I really tried to follow the mandate from on high to use Copilot, but the Agent mode can't even write code that compiles with the tools available to it.
Luckily I hooked it up to gptel so I can at least ask it quick questions about big functions I don't want to read in emacs.
This sounds fucking awesome.
Fully vibe coded, which at least they admitted. And when I pointed out the thing is off by an order of magnitude, and as such doesn't implement said feature — at all — we get pressed on our AI policy, so as to not waste their time.
I don't have an AI policy, like I don't have an IDE policy, but things get ridiculous fast with vibe coding.
Deleted Comment
You could intuitively think it's just a difference of degree, but it's more akin to a difference of kind. Same for a nuke vs a spear, both are weapons, no one argues they're similar enough that we can treat them the same way
Seriously, this cannot be emphasized enough. Before LLMS when we were writing tests completely manually, manual testing made sense to me as the second step. However after playing around a lot with coding agents and LLMs, I fully agree this has flipped. Test it manually first! When you generate the tests it is extremely wise to ensure that the tests fail without the new code, and pass with it. You definitely need to review the test though, because it's remarkably easy to have the agent put something in there that makes it not a good test.
Just a couple days ago for example, Claude made a test pass by skipping authentication and leaving a brief comment informing that the authentication made the test flaky. It even threw a quick variable in there that enabled running or disabling flaky tests, and flaky tests were disabled by default! Had I not been doing a good review, I definitely would have missed it because it was cleverly subtle. I've also seen it test the wrong endpoint!
Therefore you must verify it works as intended in the real world. This means not shipping code and hoping for the best, but checking that it actually does the right thing in production. And on top of that, you have to verify that it hasn't caused a regression in something else in production.
You could try to do that with tests, but tests aren't always feasible. Therefore it's important to design fail-safes into your code that ALERT YOU to unexpected or erroneous conditions. It needs to do more than just log an error to some logging system you never check - you must actually be notified of it, and you should consider it a flaw in your work, like a defective pair of Nikes on an assembly line. Some kind of plumbing must exist to take these error logs (or metrics, traces, whatever) and send it to you. Otherwise you end up producing a defective product, but never know it, because there's nothing in place to tell you its flaws.
Every single day I run into somebody's broken webapp or mobile app. Not only do the authors have no idea (either because they aren't notified of the errors, or don't care about them), there is no way for me to even e-mail the devs to tell them. I try to go through customer support, a chat agent, anything, and even they don't have a way to send in bug reports. They've insulated themselves from the knowledge of their own failures.
Who popped this balloon? I know I need to change my employer, but it's not so easy. And I'm not sure another employer is going to be any better.
I vibe code a lot of stuff for myself, mostly for viewing data, when I don’t really need to care how it works. I’m coming around to the idea that outside of some specific circumstances where everyone has agreed they don’t need to care about or understand the code, team vibe coding is a bad practice.
If I’m paying an engineer, it’s for their work, unless explicitly agreed otherwise.
I think vibe coding is soon going to be seen the same way as “research” where you engage an offshore team (common e.g. in consulting) to give you a rundown on some topic and get back the first five google search results. Everyone knows how to do that, if it’s what they wanted they wouldn’t be hiring someone to do it.
The second time it happens they gotta go.
I would find the expectation that I need to attach a screenshot insulting. And the understanding that my peers test their code to produce a screenshot would be pretty demoralizing.