As a programmer of over 20 years - this is terrifying.

I'm willing to accept that I just have "get off my lawn" syndrome or something.

But the idea of letting an LLM write/move large swaths of code seems so incredibly irresponsible. Whenever I sit down to write some code, be it a large implementation or a small function, I think about what other people (or future versions of myself) will struggle with when interacting with the code. Is it clear and concise? Is it too clever? Is it too easy to write a subtle bug when making changes? Have I made it totally clear that X is relying on Y dangerous behavior by adding a comment or intentionally making it visible in some other way?

It goes the other way too. If I know someone well (or their style) then it makes evaluating their code easier. The more time I spend in a codebase the better idea I have of what the writer was trying to do. I remember spending a lot of time reading the early Redis codebase and got a pretty good sense of how Salvatore thinks. Or altering my approaches to code reviews depending on which coworker was submitting it. These weren't things I were doing out of desire but because all non-trivial code has so much subtlety; it's just the nature of the beast.

So the thought of opening up a codebase that was cobbled together by an AI is just scary to me. Subtle bugs and errors would be equally distributed across the whole thing instead of where the writer was less competent (as is often the case). The whole thing just sounds like a gargantuan mess.

Change my mind.

My theory is grammatical correctness and specificity. I see a lot of people prompt like this:

"use python to write me a prog that does some dice rolls and makes a graph"

Vs

"Create a Python program that generates random numbers to simulate a series of dice rolls. Export a graph of the results in PNG format."

Information theory requires that you provide enough actual information. There is a minimum amount of work to supply the input. Otherwise, the gaps will get filled in with noise, working, what you want, or not.

For example, maybe someday you could say "write me an OS" and it would work. However, to get exactly what you want, you still have to specify it. You can only compress so far.

The most likely explanation is that the code you are writing has low information density and is stringing things together the same way many existing apps have already done.

That isn’t a judgement but trying to use the ai code completion tools for complex systems tasks is almost always a disaster.

I'm not sure how many people are like me, but my attempts to use Copilot have largely been the context of writing code as usual, occasionally getting end-of-line or handful-of-lines completions from it. I suspect there's probably a bigger shift needed, but I haven't seen anyone (besides AI "influencers" I don't trust..?) showing what their day-to-day workflows look like.

Is there a Vimcasts equivalent for learning the AI editor tips and tricks?

I agree with you and its confusing to me. I do think there is a lot of emotion at play here - rather than cold rationality.

Using LLM based tools effectively requires a change in workflow that a lot of people aren't ready to try. Everyone can share their anecdote of how an LLM has produced stupid or buggy code, but there is way too much focus on what we are now, rather than the direction of travel.

I think existing models are already sufficient, its just we need to improve the feedback loop. A lot of the corrections / direction I make to LLM produced code could 100% be done by a better LLM agent. In the next year I can imagine tooling that: - lets me interact fully via voice - a separate "architecture" agent ensures that any produced code is in line with the patterns in a particular repo - compile and runtime errors are automatically fed back in and automatically fixed - a refactoring workflow mode, where the aim is to first get tests written, then get the code working, and then get the code efficient, clean and with repo patterns

I'm excited by this direction of travel, but I do think it will fundamentally change software engineering in a way that is scary.

I guess for me it actually takes longer to review code than to write it. So maybe that’s some of the difference.

If you're doing something that appears in it's training model a lot, like building a twitter clone, then it is great. If you're using something brand new like react router 7 then it makes mistakes

I think it's bimodal because there's a roughly bimodal distribution of high level attitudes among programmers. There's one clump that are willing to be humble and interact with the AI in a thoughtful, careful manner, acknowledging that it might be smarter than them (e.g. see Terry Tao's comments regarding mathematics usage about how in order to get good results he takes care with what he puts in (and imagine what "care" means for a professional mathematician!)) and there's another clump who aren't.

>My theory is the willingness to baby sit and the modality. I'm perfectly fine telling the tool I use its errors and working side by side with it like it was another person.

In my experience, baby sitting the AI takes to much time and effort. I'd rather do it myself and use AI for tasks I don't have to babysit.

> Im actually very curious why AI use is such a bi-modal experience. I've used AI to move multi thousand line codebases between languages. I've created new apps from scratch with it.

I think this depends on the nature of your work. I've been successful with using LLMS for creating things from scratch, for myself, especially in a domain I was not familiar with and am quite happy with that. Things like proof of concepts or exploring a library or a framework. But in my current work setting, relying on LLMS to do production work is only somewhat helpful here and there but nowhere near as helpful as in the first case. In some cases it hallucinated so close to what it was supposed to do that it introduced a bug I would have never created had I not used LLMs and that took a lot of effort to spot.

> I've created new apps from scratch with it.

I don't think that "creating new apps from scratch" should be the benchmark. Unless you're doing something very novel, creating a new app/service is rather formulaic. Many frameworks even have templates / generators for that sort of thing. LLMs are maybe just better generators - which is not useless, but it's not where the real complexity of software development lies.

The success stories I am looking for are things like "I migrated a Java 6 codebase with a legacy app server to Java 21", "I ripped out this unsupported library across the project and replaced it with a better one" or "I refactored the codebase so that the database access is in its own layer". If LLMs can do those tasks reliably, then I'll have another look.

> I'm actually very curious why AI use is such a bi-modal experience

I think it's just that it's better at some things than others. Lucky for people who happen to be working in python/node/php/bash/sql/java probably unlucky for people writing Go and Rust (I'm hypothesising because I don't know Go or Rust nor have I ever used them but when the AI doesn't know something it REALLY doesn't know it, like it goes from being insanely useful to utterly useless).

> I use AI autocomplete 0% of the time as I found that workflow was not as effective as me just writing code, but most of my most successful work using AI is a chat dialogue where I'm letting it build large swaths of the project a file or parts of a file at a time, with me reviewing and coaching.

Me too, the way I use it is more like pair programming.

MMV But for me at least i tend to use it for brain storming, aka intial sailing through a subject/topic/task, getting intial idea. the idea is to use as an admin who is guided by you throgh chatting. for example im given a task to translate a user description/requirement to pull something from the database. like (simplistic example) what are the top grossing films by category within each rating. so igive the AI the database tables schema and give it literally the user requirement. and see what it gave back and compare it with how I'll do it. ask it more for optimizations what else can be done more.... etc.. keep chating with the AI until I'm bored ;)

I'm curious coming from the other end. I guess I can totally understand certain use cases where I'm generating fairly simple, self contained code in a language I'm unfamiliar with being good.

But surely you must have experienced something where you're literally fighting with the model, where it continuously repeats its mistakes, and fixing a mistake in one place, breaks something else, and you can't seem to escape this loop. You then get desperate, invoking magic phrases like "you think through your problems step by step", or "you are a senior developer", only for it to loose the entire thread of the conversation.

Then the worst part is when you finally give up, your mental state of the problem is no better than when you first started off.

> I've used AI to move multi thousand line codebases between languages.

And are you certain you’ve reviewed all use cases to make sure no errors were introduced?

I recently tried using Google’s AI assistant for some basic things like creating a function that could parse storage size in the format of 12KB, or 34TB into an actual number of bytes. It confidently gave me amount, units = s.split() which just is not correct. Even added a comment explaining what that line is meant to do.

This was an obvious case that just didn’t work. But imagine it did work but flew into an infinite loop on an input like “12KB7” or some such.

I'm convinced what we are witnessing is that there are genius level engineers (lots of them) that are and many have always been sub-par communicators. I think being a good communicator tracks really well with how much someone can get out of LLMs (as does engineering competency. You need both).

Great but not genius engineers who are also great communicators may broadly outperform people with only technical genius soon, but that's speculation on my part.

> ...and I can review code very quickly so the overall productivity boost has been great.

Color me skeptical. After a certain point, greater speed is achieved by sacrificing accuracy and comprehension. So, "I can review code very quickly" starts to sound like "I don't read, I skim."

IMHO, reviewing code is one of the parts of the job that sucks, so I see "AI" as a wonderful technology to improve our lives by replacing fun with chores.

+++1

THIS!

exactly my style of working and how i think about that: i've also not enabled/installed CoPilot or similar, just AutoSuggestion by default in VS.NET But i use LLM heavily to get rid off all the exhausting tasks, and to generate ideas what to improve in some larger code blocks so i dont have to rewrite/refactor it on my own.

It boosts my productivity by 10x at least.

Interesting that you find the conversational approach effective. For me, I'd say 9 out of 10 code conversations get stuck in a loop with me telling the AI the next suggested iteration didn't actually change anything or changed it back to something that was already broken. Do you not experience that so often, of do you have a way to escape that?

> I'm perfectly fine telling the tool I use its errors and working side by side with it like it was another person.

This is key. Traditional computing systems are deterministic machines, but AI is a probabilistic machine. So the way you interact and the range, precision, and perspective of the output stretches over a different problem/solution space.

I've tried using AI(Claude) to do refactors/move code between languages, and in my experience, it has the tendency to go off the rails and just start making up code that does something similar, essentially doing a rewrite that never works.

> move multi thousand line codebases between languages

i am more curious about why someone do would this

I agree. I am in a very senior role and find that working with AI the same way you do I am many times more productive. Months of work becomes days or even hours of work

>I've created new apps from scratch with it.

Would mind sharing those apps to view. Not the code, just the apps. I have a suspicion about the bi-modal experience.

> I'm actually very curious why AI use is such a bi-modal experience.

My conspiracy theory is that the positive experiences are exaggerated and come from investors in the Nvidia stock.

>> The apparent speed up is mostly a deception.

When I am able ask a very simple question of an LLM which then prevents me having to context-switch to answer the same simple question myself; this is a big time saver for me but hard-to-quantify.

Anything that reduces my cognitive load when the pressure is on is a blessing on some level.

That’s the thing, isn’t it? The craft of programming in the small is one of being intimate with the details, thinking things through conscientiously. LLMs don’t do that.

Exactly, 1 step forward, 1 step backward. Avoiding edge cases is something that can't be glossed over, and for that I need to carefully review the code. Since I'm accountable for it, and can't skip this part anyway, I'd rather review my own than some chatbot's.

Why aren't you writing unit tests just because AI wrote the function? Unit tests should be written regardless of the skill of the developer. Ironically, unit tests are also one area where AI really does help move faster.

High level design, rough outlines and approaches, is the worst place to use AI. The other place AI is pretty good is surfacing api call or function calls you might not know about if you're new to the language. Basically, it can save you a lot of time by avoiding the need for tons of internet searching in some cases.

I would love to find out

where programmers are learning this idea:

that writing code fast is ideal.

If it takes 30 years to write one loc, it takes 30 years.

Ideally, it takes 30 years to write zero lines of code.

Reminds me of how I use the satnav when driving.

I don't close my eyes and do whatever it tells me to do. If I think I know better I don't "turn right at the next set of lights" I just drive on as I would have before GPS and eventually realise that I went the wrong way or the satnav realises there was a perfectly valid 2nd/3rd/4th path to get to where I wanted to go.

Yes, for simple boring tasks like converting a JSON to C# classes, it does wonders.

For everything clever it kind of needs baby sitting which doesn't make it faster than writing the code myself.

Even for the glorified use case of writing unit tests it sucks unless the code is very simple with few dependencies.

One off scripts do work very well.

Totally agree that heavy commenting is the best convention for helping the assistant help you best. I try to comment in a way that makes a file or function into a "story" or kind of a single narrative.

Same here. If you need to lookup how to do something in an api I find it much faster to use chatgpt than to try to search through the janky official docs or in some github examples folder. Chatgpt is basically documentation search 2.0.

The problem with this is that you lose the entire context around the answer you're given. There might be notes about limitations or edge cases in the documentation or on StackOverflow. I already see this with my coworkers who use LLMs, they often have no idea how or why things work and then they're absolutely perplexed when something goes wrong.

I love your framing of it as a "better stackoverflow." That's so true. However, I feel like some of our complaints about accuracy and hidden bugs are temporary pain (12-36) months before the tools truly become mind-blowing productivity multipliers.

>"better stackoverflow"

I had asked hard questions on SO that only a human with experience can answer. And I found answers on SO that only a human with experience can answer.

> How can this be possible if you literally admit its tab completion is mindblowing?

I might suggest that coding doesn't take as much of our time as we might think it does.

Hypothetically:

Suppose coding takes 20% of your total clock time. If you improve your coding efficiency by 10%, you've only improved your total job efficiency by 2%. This is great, but probably not the mind-blowing gain that's hyped by the AI boom.

(I used 20% as a sample here, but it's not far away from my anecdotal experience, where so much of my time is spent in spec gathering, communication, meeting security/compliance standards, etc).

> How can this be possible if you literally admit its tab completion is mindblowing?

What about it makes it impossible? I’m impressed by what AI assistants can do - and in practice it doesn’t help me personally.

> Select line of code, prompt it to refactor, verify they are good, accept the changes.

It’s the “verify” part that I find tricky. Do it too fast and you spend more time debugging than you originally gained. Do it too slow and you don’t gain much time.

There is a whole category of bugs that I’m unlikely to write myself but I’m likely to overlook when reading code. Mixing up variable types, mixing up variables with similar names, misusing functions I’m unfamiliar with and more.

If someone can eat 20 golf balls that’s impressive but it doesn’t improve my golf game

> How can this be possible if you literally admit its tab completion is mindblowing?

If I had a knife of perfect sharpness which never dulled, that would be mind-blowing. It also would very likely not make me a better cook.

In experience, I always have to spend time to check and most times it doesn’t do what I need unless it’s very simple asks.

Out of curiosity, how long have you been working as a developer? Just that, in my experience, this is mostly true for juniors and mids (depending on the company, language, product etc. etc.). For example, I often find that copilot will hallucinate tailwind classes that don't exist in our design system library, or make simple logical errors when building charts (sometimes incorrect ranges, rarely hallucinated fields) and as soon as I start bringing in 3rd party services or poorly named legacy APIs all hope is lost and I'm better off going it alone with an LSP and a prayer.

> SpaceX's advancements are impressive, from rocket blow up to successfully catching the Starship booster.

That felt like it was LLM generated since that doesn't have anything to do with the subject being discussed. Not only it's on a different industry but it's a completely different set of problems. We know what's involved in catching a rocket. It's a massive engineering challenge yes, but we all know it can be done(whether or not it makes sense or is economically viable are different issues).

Even going to the Moon – which was a massive project and took massive focus from an entire country to do – was a matter of developing the equipment, procedures, calculations (and yes, some software). We knew back then it could be done, and roughly how.

Artificial intelligence? We don't know enough about "intelligence". There isn't even a target to reach right now. If we said "resources aren't a problem, let's build AI", there isn't a single person on this planet that can tell you how to build such an AI or even which technologies need to be developed.

More to the point, current LLMs are able to probabilistically generate data based on prompts. That's pretty much it. They don't "know" anything about what they are generating, they can't reason about it. In order for "AI" to replace developers entirely, we need other big advancements in the field, which may or may not come.

Except cursor is the fireworks based on black powder here. It will look good, but as a technology to get you to the moon it seems to look like a dead end. NOTHING (of serious science) seems to indicate LLMs being anything but a dead end with the current hardware capabilites.

So then I ask: What, in qualitative terms, makes you think AI in the current form will be capable of this in 5 or 10 years? Other than seeing the middle of what seems to be an S-curve and going «ooooh shiny exponential!»

> or even replace developers

I don't think there will be a 'replace developers, but other work remains extant' moment - not at least for very long at all.

>If this is how you use Cursor then you dont need Cursor.

I've tried Cursor yesterday only to realise it can't run and debug C# projects.

I think SuperMaven has tab complete and let's you chose the model.

In what way exactly? I wish Zed was better here but it's at about the same level as the current nvim plugins avail. As far as I can tell it's just a UI for adding context.

It’s all a dice game with these things, you have to watch them closely or they start running you (with bad outcomes). Disclaimers aside:

Sonnet is better in the small, by a lot. It’s sharply up from idk, three months ago or something when it was still an attractive nuisance. It still tops out at “Best SO Answer”, but it hits that like 90%+. If it involves more than copy paste, sorry folks, it’s still just really fucking good copy paste.

But for sheer “doesn’t stutter every interaction at the worst moment”? You’ve got to hand it to the ops people: 4o can give you second best in industrial quantity on demand. I’m finding that if AI is good enough, then OpenAI is good enough.

Are there any good 3rd-party native frontend apps for Claude (on MacOS)? I mean something like ChatGPTs app, not an editor. I guess one option would be to just run Claude iPad app on MacOS.

> But OpenAI provides a much more capable app in ChatGPT and an easier development platform

Which app are you talking about here?

FWIW, I was able to get a decent way into making my own client for ChatGPT by asking the free 3.5 version to do JS for me* before it was made redundant by the real app, so this shouldn't be too hard if you want a specific experience/workflow?

* I'm iOS by experience; my main professional JS experience was something like a year before jQuery came out, so I kinda need an LLM to catch me up for anything HTML

Also, I wanted HTML rather than native for this.

> ChatGPT as an app is just so far ahead: code interpreter, web search/fetch, fluid voice interaction, Custom GPTs, image generation, and memory. It isn't close.

Funny thing, TypingMind was ahead of them for over a year, implementing those features on top of the API, without trying to mix business model with engineering[0]. It's only recently that ChatGPT webapp got more polished and streamlined, but TypingMind's been giving you all those features for every LLM that can handle it. So, if you're looking for ChatGPT-level frontend to Anthropic models, this is it.

ChatGPT shines on mobile[1] and I still keep my subscription for that reason. On desktop, I stick to TypingMind and being able to run the same plugins on GPT-4o and Claude 3.5 Sonnet, and if I need a new tool, I can make myself one in five minutes with passing knowledge of JavaScript[2]; no need to subscribe to some Gee Pee Tee.

Now, I know I sound like a shill, I'm not. I'm just a satisfied user with no affiliation to the app or the guy that made it. It's just that TypingMind did the bloodingly stupid obvious thing to do with the API and tool support (even before the latter was released), and continues to do the obvious things with it, and I'm completely confused as to why others don't, or why people find "GPTs" novel. They're not. They're a simple idea, wrapped in tons of marketing bullshit that makes it less useful and delayed its release by half a year.

--

[0] - "GPTs", seriously. That's not a feature, that's just system prompt and model config, put in an opaque box and distributed on a marketplace for no good reason.

[1] - Voice story has been better for a while, but that's a matter of integration - OpenAI putting together their own LLM and (unreleased) voice model in a mobile app, in a manner hardly possible with the API their offered, vs. TypingMind being a webapp that uses third party TTS and STT models via "bring your own API key" approach.

[2] - I made https://docs.typingmind.com/plugins/plugins-examples#db32cc6... long before you could do that stuff with ChatGPT app. It's literally as easy as it can possibly be: https://git.sr.ht/~temporal/typingmind-plugins/tree. In particular, this one is more representative - https://git.sr.ht/~temporal/typingmind-plugins/tree/master/i... - PlantUML one is also less than 10 lines of code, but on top of 1.5k lines of DEFLATE implementation in JS I plain copy-pasted from the interwebz because I cannot into JS modules.

Have you tried using Cursor with Claude embedded? I can't go back to anything else, it's very nice having the AI embedded in the IDE and it just knows all the files i am working with. Cursor can use GPT-4o too if you want

I'm the same, but had a lot of issues getting structured output from Anthropic. Ended up always writing response processors. Frustrated by how fragile that was, decided to try OpenAI structured outputs and it just worked and since they also have prompt caching now, it worked out very well for my use case.

Anthropic's seems to have addressed the issue using pydantic but I haven't had a chance to test it yet.

I pretty much use Anthropic for everything else.

>The second is with all the OpenAI drama MS has started to distance themselves over a souring relationship (allegedly). If so, this could be a smart move away tactfully.

I agree, this was a tactical move designed to give them leverage over OpenAI.

Yeah, Claude consistently impresses me.

A commenter on another thread mentioned it but it’s very similar to how search felt in the early 2000s. I ask it a question and get my answer.

Sometimes it’s a little (or a lot) wrong or outdated, but at least I get something to tinker with.

Yes! I'm much more inclined to write one-off scripts for short manual tasks as I can usually get AI to get something useful very fast. For example, last week I worked with Claude to write a script to get a sense of how many PRs my company had that included comprehensive testing. This was borderline best done as a manual task previously, now I just ask Claude to write a short bash script that uses the GitHub CLI to do it and I've got a repeatable reliable process for pulling this information.

Lots of progress, but I feel like we've been seeing diminishing returns. I can't help but feel like recent improvements are just refinements and not real advances. The interest in AI may drive investment and research in better models that are game-changers, but we aren't there yet.

I wonder how long people will still protest in these threads that "It doesn't know anything! It's just an autocomplete parrot!"

Because.. yea, it is. However.. it keeps expanding, it keeps getting more useful. Yea people and especially companies are using it for things which it has no business being involved in.. and despite that it keeps growing, it keeps progressing.

I do find the "stochastic parrot" comments slowly dwindle in number and volume with each significant release, though.

Still, i find it weirdly interesting to see a bunch of people be both right and "wrong" at the same time. They're completely right, and yet it's like they're also being proven wrong in the ways that matter.

Very weird space we're living in.

I was at the keynote, Llama was featured in the Copilot models section and called out specifically, as was Mistral.

I assume they just aren't at the point where they have the ability or want to host the compute to offer up Llama as an option as opposed to OpenAI, Anthropic and Google who are all offering the model as a service.

Try to fight the free good-enough haha. At least that’s the plan of Meta, who does not benefit as much selling this than using this

Or, anti trust fears.

Your vscode and rider integrations are fantastic, love the different ways to add context to the chat

It's Mona Lisa the Octocat: https://github.com/monatheoctocat

It’s October 30th in several parts of the world already. It’s after midnight everywhere GMT+7 onwards.

It says the 29th now