This has been about 50% of the time my experience as well. There are very good SWE who know how to use ML in real systems, and then there are the others who believe through and through it will replace well understood systems developed by subdomain experts.

As a concrete example, when I worked at Amazon, there were several really good ML-based solutions for very real problems that didn't have classical approaches to lean on. Motion prediction from grid maps, for example, or classification from imagery or grid maps in general. Very useful and well integrated in a classical estimation and control pipeline to produce meaningful results.

OTOH, when I worked at a startup I won't name, I was berated over and over by a low-level manager for daring to question a learning-based approach for, of all things, estimating orientation of a stationary plane over time. The entire control pipeline for the vehicle was being fed flickering, jumping, adhoc rotating estimate for a stationary object because the entire team had never learned anything fundamental about mapping or filtering, and was just assuming more data would solve the problem.

This divide is very real, and I wish there was a way to tease it out better in interviewing.

> So, for MLE, working with AI that isn't always reliable, is a norm. They are accustomed to thinking in terms of probabilities, distributions, and acceptable levels of error. Applying this mindset to a coding assistant that might produce incorrect or unexpected code feels more natural. They might evaluate it like a model: "It gets the code right 80% of the time, saving me effort, and I can catch the 20%."

And given the current climate, the MLE's feel empowered for force their mindset onto others groups where it doesn't fit. I once heard a senior architect at my company ranting about that after a meeting: my employer sells products where accuracy and correctness have always been a huge selling point, and the ML people (in a different office) didn't seem to get that and thought 80-90% correct should be good enough for customers.

I'm reminded of the arguments about whether a 1% fatality rate for a pandemic disease was small or large. 1 is the smallest integer, but 1% of 300 million is 3 million people.

You're talking about deterministic behavior vs. probabilistic behavior and yes some discourse lines up with what you describe.

I don't think it's the case with this article. It focuses on the meta-concerns of people doing software engineering and how AI fits into that. I think he hits it on the head when he talks about Program Entropy.

A huge part of building a software product is managing entropy. Specifically, how you can add more code and more people while maintaining a reasonable forward velocity. More specifically, you have to maintain a system so you make it so all of those people understand how all the pieces fit together and how to add more of those pieces. Yes, I can see AI one day making this easier but right now, it oftentimes makes entropy worse.

There are so many use cases where 90% correct answers are absolutely not enough. Nobody would have much of a problem with that, if a flurry of people with vested interests wouldn't try to convince us all that that is not the case, and AI is good to go for absolutely everything. The absurdity of this assumption is so outrageous that it becomes even hard to counter it with logic. It's just a belief-based narrative whose delivery has been highly successful so far in commanding insane investments, and as a travestee for profit-oriented workforce optimizations.

SWE use probability all the time. Rearchitect around that race condition or reduce its footprint? How long will this database call make, p99? A/B tests. Etc.

I’m currently getting my masters in AI (lots of ML) and as a SWE, it’s definitely a new muscle I’m growing. At the same time, I can think about MLE in isolation and how it fits within the larger discipline of SWE. How I can build robust pipelines, integrate models into applications, deploy models within larger clusters, etc. I think there are many individuals which are pure MLE and lack the SWE perspective. Most critically, lots of ML people in my program aren’t computer people. They are math people or scientists first. They can grok the ML but grokking SWE without computer affinity is difficult. I see true full-stack being an understanding of low-level systems, back-front architecture, deployment, and now MLE. Just need to find someone who will compensate me for bringing all that to the table. Most postings are still either for SWE or PhD in MLE. Give me money!! I know it all

Yea, the problem is that most people expect things to be absolute and correct outside of engineering.

I love the gray areas and probabilities and creativity of software...but not everyone does.

So the real danger is in everyone assuming the AI model is, must be, and always will be correct. They misunderstand the tool they are using (or directing others to use).

Hmm. It's like autopilot on the Tesla. You aren't supposed to take your hands off the wheel. You're supposed to pay attention. But people use it incorrectly. If they get into an accident, then people want to blame the machine. It's not. It's the fault of person who didn't read the instructions.

I often wonder if society will readjust its expectation of programs or even devices. Historically, machines of all kinds were difficult to design and manufacture.. the structure was hard set (hence the name) but at the same time, society fantasize about adaptive machines, hyper adaptive, multipurpose, context-aware.. which if pushed high, is not far from the noisy flexibility of ML.

Yup. All the people I've worked with through my career post 2020 (AI/ML types) have been AI first and AI native. They're the first users of cursor - a year before it went mainstream.

Sorry not sorry that the rest of the world has to look over their shoulders.

Based on my experience as an MLE I would never use any of the current 'AI' offerings, so whatever bias you are suggesting is very recent.

i agree; but perhaps also it is the difference between managers and SWE? The former (SWE team leaders included) can see that engineers aren't perfect. The latter are often highly focused on determinism (this works/doesn't) and struggle with conflicting goals.

Through a career SWEs start rigid and overly focused on the immediate problem and become flexible/error-tolerant[1] as they become system (mechanical or meat) managers. this maps to an observation that managers like AI solutions - because they compare favourably to the new hire - and because they have the context to make this observation.

[1] https://grugbrain.dev/#:~:text=grug%20note%20humourous%20gra...

I appreciate your nuanced position. I believe that any developer who isn't reading more code than they are writing is doing it wrong. Reading code is central to growth as a software engineer. You can argue that you'll be reading more bland code when reviewing code generated with the aid of an LLM. I still think you are learning. I've read lots of LLM generated code and I routinely learn new things. Idioms that I wasn't familiar with, or library calls I didn't know existed.

I also think that LLMs are an even more powerful accelerant for senior developers. We can prompt better because we know what exists and what to not bother trying.

I don't think it is becoming a series of code reviews, more like having something do some prototyping for you. It is great for fixing the blank page problem, but not something you can review and commit as is.

Problem is, paraphrasing Scott Kilmer, corporations are dead from the neck up. The conclusion for them was not that AI will help juniors, is that they will not hire juniors and will ask seniors the magic "10x" with the help of AI. Even some seniors are getting the boot, because AI.

Just look at recent news, layoff after layoff from Big Tech, Middle tech and small tech.

Great analogy. 3d printing is awesome and incredibly useful tech. Truly world changing. But injection molding is here to stay.

It might not lead to singularity but for people who work in academia, in terms of setting and marking assignments and lecture notes, for good or bad AI has had an enormous impact.

You might argue that LLMs have simply exposed some systematic defects instead of improving anything, but the impact is there. Dozens of lecturing workflows that were pretty standard 2 years ago are no longer viable. This includes the entirety of online and remote education which ironically dozens of universities started investing in after Covid, right around when chatgpt launched. To put this impact in context, we are talking about the tertiary and secondary sector globally.

This is an easy quip to make, but it's also pretty wrong. 3D printing has been a massive breakthrough in many industries and fundamentally changed the status quo. Aerospace is a good example, much of what SpaceX and other younger upstarts in the space are doing would not be feasible without 3D printed parts. Nozzles, combustion chambers, turbopumps etc are all parts that are often printed.

or bitcoin going to replace banks? We ended up with banks selling financial tools based on bitcoin.

I honestly don’t, although maybe that hype cycle was before my time.

But this seems an unfair comparison. For one, I think 3D printing made me better, not worse, at engineering (back when mechanical engineering was my jam), as it allowed me to prototype and make mistakes faster and cheaper. While it hasn’t replaced all manufacturing (or even come close), it plays an important role in design without atrophying the skills of the user.

Well yeah, the singularity isn't close by any measure.

But 3D printing and AI are on totally different trajectories.

I haven't heard of Mattel saying, "we're going to see in what places we can replace standard molding with 3d printing". It's never been considered a real replacement, but rather potentially a useful way to make PoCs, prototypes and mockups.

I have a 3d printer and cadded me some parts, I don't have an injection molding plant.

3d printing will slowly edge its way into more manufacturing. The humble stepping motor really is eating the world. 3dp is one manifestation of it!

Back to AI though.

I just checked the customer support page of a hyped AI app generator and its what you expect: "doesn't work on complex project" "wastes all my tokens" and "how to get a refund"

These things are over promising and a future miracle is required to justify valuations. Maybe the miracle will come maybe not.

> Remember when 3d printing was going to replace all manufacturing? Anybody?

Sure, but I'd argue the AIs are the new injection molding (as mentioned downthread) with the current batch being the equivalent of Bakelite.

Plus, who seriously said 3d printers were going to churn out Barbies by the millions? What I remember is people claiming they would be a viable source of one-off home production for whatever.

Not saying that it’s not true but hardware and software have different trajectories.

for small run manufacturing, 3d printing is absolutely killing it. the major limitation of 3d printing is that it will never be able to crank out thousands of parts per hour the way injection molding can and thats ok. creating an injection molded part requires a huge up front investment. if you're doing small runs of a part, 3d printing more than makes up for the slow marginal time with skipping the up front costs alltogether.

Or 3d cinema. Or vr. All just hype bs

No, because 3D printing was never going to replace all manufacturing. Anyone who said that didn't even have a basic understanding of manufacturing, and I don't recall any serious claims like that.

3D printing has definitely replaced some manufacturing, and it has had a huge effect on product design.

These anti-AI articles are getting more tedious than the vibe coding articles.

> is borrowed comprehension,

Or would be, if the LLM actually understood what it was writing, using the same definition of understanding that applies to human engineers.

Which it doesn't, and by its very MO, cannot.

So, every line from an LLM that is accepted without understanding, is really nonexistent comprehension. It's a line of code, spat out by a stochastic model, and until some entity that actually can comprehend a codebases context, systems and designs (and currently the only known entity that can do that is a human being), it is un-comprehended.

Depending on the task they are terrible at writing code.

This is a very good analogy. And this interest rate can probably be significantly reduced by applying TDD and reducing the size of isolated subsystems. That may start to look like microservices. I generally don’t like both for traditional development, but current LLMs both make them easier and more useful.

And the “rule of three” basically ceases to be applicable between components — either the code has localized impact, or is a part of rock-solid foundational library. Intermediate cases just explode the refactoring complexity.

From my experience so far it is helpful for me to get another opinion on how to solve a problem—and I do the work in the end. Or, I am extremely specific, and give it a relatively small problem to solve, and it solves it—writes the code for me—and then I code review it, and make changes to uphold my standards.

In other words, AI is my assistant, but it is MY responsibility to turn up quality, maintainable work.

However, to put things in perspective for the masses: just consider the humble calculator. It has ruined people’s ability to do mental math. AI is going to do that for writing and communication skills, problem solving skills, etc.

> but a single word used in a vague enough way is enough to skew the results in a bad direction

I'm glad I'm not the only one who feels this way. It seems like these models latch on to a particular keyword somewhere in my prompt chain and throw traditional logic out the window as they try to push me down more niche paths that don't even really solve the original problem. Which just leads to higher levels of frustration and unhappiness for the human involved.

> Anecdotally, I've felt my skills quickly regressing because of AI tooling

To combat this, I've been trying to use AI to solve problems that I normally would with StackOverflow results: for small, bite-sized and clearly-defined tasks. Instead of searching "how to do X?", I now ask the model the same question and use its answer as a guide to solving the problem instead of a canonical answer.

Definitely share your feeling that people move the goalposts from "AI can do it" to "well it would have been able to do it if you used model o2.7 in an IDE with RAG and also told it how to do it in the prompt" ...ok, at some point it's less value for the effort than writing the code myself, thanks

That said, AI does make some things easier today, like if you have an example to use for "make me a page like this but with data from x instead of y". Often it's faster than searching documentation, even with the caveat that it might hallucinate. And ofc it will probably improve over time.

The particular improvement I'd like to see is (along with in general doing things right) finding the simplest solution without constantly having to be told to do so. My experience is the biggest drawback to letting chatgpt/claude/etc loose is quickly churning out a bunch of garbage, never stopping to say this will be too complex to do anything with in the future. TFA claims only humans can resist entropy by understanding the overall design; again idk if that will never improve but it feels like the big problem right now.

What I've been doing when I want to avoid this "unexpected leading", is to tell the LLM to "Ask me 3 rounds of 5 clarifying questions each, first.". The first round usually exposes the main assumptions it's making, and from there we narrow down and clarify things.

I've read you comment about all the things you tried, and it seems you have much broader experience with LLMs than I do. But I didn't see this technique mentioned, so leaving this here in case it helps someone else :).

People have similar biases, it's just a lot easier to test an LLM for biases than it is to test humans so we are more aware of LLM biases.

I consider this hype the preparatory phase for the era where we rediscover the laws of quality control.

A GREAT example is good old Coke vs Pepsi.

Almost all of ai hype seems to be on the backs of no coders or people who make toys as a career (if they even have one)

Nonono you've got it all wrong you clearly need to try [model name] with [coding product], it 100x'd my productivity!

You'd think the whole "LLMs can't reason in concepts" meme would've died already. LLMs are literally concepts incarnate, this has already been demonstrated experimentally in many ways, not limited to figuring out how to identify and suppress or amplify specific concepts during inference.

Article also repeats some weird arguments that are superficially true, but don't stand to scrutiny. That Naur thing, which is a meme at this point, is often repeated as somehow insightful in the real world - yet what's forgotten is another fundamental, practical rule of software engineering: any nontrivial program quickly exceeds any one's ability to hold a full theory of it in their head; we almost never work with proper program theory; programming languages, techniques, methodologies and tools all evolve towards enabling people to work better without understanding most of the code. We actually share the same limitations as LLMs here, we're just better at managing it because we don't have to wait for anyone to let us do another inference loop so we can take a different perspective.

Etc.

A big problem I keep facing when reviewing junior engineers code is not the code quality itself but the direction the solution went into, I'm not sure if LLM models are capable of replying to you with a question of why you want to do it that way(yes like the famous stackoverflow answers).

> I'm also having a great time converting code into [...] diagrams

Do you do it manually or a have automated tool? (I am looking for the latter.)

> I think you could make similar arguments about mapping technology like Google and Apple Maps

The problem is that mapping software is reliable and doesn't spit out a result of what is essentially a random number generator. You can rely on its output, the same way you can rely on a calculator. Not always, mind you, because mapping the entire globe is a massively complex task with countless caveats and edge cases, but compared to LLM output? Even with a temperature setting of 0 with the same prompt regenerated multiple times, you'll be getting vastly different output.

Also, since LLMs cover a much more broad swathe of concepts, people are going to be using these instead of their brains in a lot of situations where they really shouldn't. Even with maps, there are people out there that will drive into a lake because Google Maps told them that's where the street was, I can't even fathom the type of shit that's going to happen from people blindly trusting LLM output and supplanting all their thinking with LLM usage.

> I think you could make similar arguments about mapping technology like Google and Apple Maps -- that using them decreases people's skills in navigating the physical world, atrophying our sense of direction and geography. And actually, that's not wrong. People really do often struggle to navigate these days if they don't have the crutch of something like Google Maps. It really has changed our relationship to the physical world in many ways

Entirely anecdotal but I have found the opposite. With this mapping software I can go walk in a random direction and confidently course correct as and when I need to, and once I’ve walked somewhere the path sticks in my memory very well.

The reliability difference is so big that the analogy isn’t relevant.

Google maps is 90% of the time better than a taxi driver where I live.

AI isn’t better than some person that did the thing for a couple days

> The overall average ability of people being able to get from Point A to Point B safely and reliably, especially in areas they are unfamiliar with, has certainly increased dramatically.

Is there evidence for this?

That's an extremely self-serving narrative. I assume you're part of the 30 percent?

It's almost like people outsourcing their job to AI are asking to get fired, not only by proving that a computer program can do their job better, but even paving the way for it!

For large companies this is true, and this may be the best AI coding take I have read.

It's similar with full self drive. FSD is better than a bad, drunk, or texting human driver, and that's a lot of the drivers on the road.