One consideration not mentioned is around developer sophistication. Steve alludes to the expansion effect of CodeGen ("there are millions and maybe billions who are jumping at the chance to code"), but doesn't consider that the vast majority of these people don't know about arrays, data structures, memory, containers, runtimes, etc, etc...
To me, that's the most important consideration here. Are you targeting professional devs who are enhancing their current workflows iteratively with these improvements? Or re-thinking from the ground up, obfuscating most of what we've learned to date?
Maybe we need to trudge through all of these weeds until software creation hits its final, elegant form where "Anyone Can Code".
Maybe the old Gusteau quote is actually fitting here:
"You must be imaginative, strong-hearted. You must try things that may not work, and you must not let anyone define your limits because of where you come from. Your only limit is your soul. What I say is true - anyone can ̶c̶o̶o̶k̶ code... but only the fearless can be great."
Well we'll never reach a state where anyone can code. I have pans, a supermarket nearby, cookbooks and a belly, still I'm never gonna be able to cook, I snooze after 30 minutes, even if I succeed once, I get bored and stop for months etc.
Simplifying to the point a grandma could make an app isn't gonna make any grandma WANT to make apps. And that's fine, there's no issue, we don't have to make more people code and those who want, will, even if all we had was assembly and a light board...
Which I think is the spirit of your quote basically.
That’s a bad comparison, cooking has been done by people for thousands of years, your problem with cooking is laziness, there is nothing mentally or physically stopping your from learning to cook.
I do agree with your second paragraph and it’s more that you DON’T want to cook versus you being unable to cook.
Windsurf + Haskell w/ CLI tools has been pretty amazing. Windsurf's agent will loop for minutes on its own to figure out the right structure of a program. You just need to tell it to:
- use the hoogle cli to search for the right types and functions
Woah, that sounds awesome! I'd love to see how you set that up and how much it can do without your intervention/approval for various actions. Might you have a video of your workflow that you could share?
What's the maximum file size for which this is useful in your experience? I have been refactoring some project solely to enable AI code editors to edit it. Some users in the discord suggest a maximum file size of 500LOC or small, which seems unreasonable.
> The next big thing was Cursor. I must admit that I never personally fell in love with it, but given how many people I respect love it, I think that’s a me-problem
I've met so many engineers who have said exactly this. There are clearly some group of people obsessed with Cursor, but it's interesting to me how alien they seem to the majority of people using ai codegen right now.
i had to uninstall it because it had associated itself with every possible file extension. i couldn't open a file without cursor popping up. very horrifying for that to happen to my computer when working on important projects
On Linux, I have the opposite issue. I ended up hard symlinking cursor to VS Code because Cursor wasn't opening despite being set as the default editor.
I prefer vscode+copilot. It's much cheaper and has all the functionality I want. There's access to 3.5 sonnet, and it can edit/create up to 10 files at a time.
I'm pretty much full time on cursor from vscode. I don't trust it for big code blocks, but a control-k + reasonable command (I could have typed up myself) is saving me quite a bit of time.
This blog article is written in a very engaging way. It seems to be more or less a masterclass on how to keep someone's attention, although there is no meta-story making you wait for the big fulfillment at the end.
I think it is the short, punchy sections with plenty of visuals and the fact that you are telling a story the whole way through, which has a natural flow, each experiment you describe, leading to the next.
For the lmarena leaderboard to be really useful you need click the "Style Control" button so that it normalizes for LLMs that generate longer answers, etc. that, while humans may find them more stylistically pleasing, and upvote them, the answers often end up being worse. When you do that, o1 comes out on top followed by o1-preview, then Sonnet 3.5, and in fourth place Gemini Preview 1206.
It's really interesting seeing the progression here, integrating AI-assisted coding tools into something like Val Town is a great arena for exploring different patterns for this stuff.
Worth checking out their Cerebras-powered demo too - LLMs at 2000 tokens/second make applying proposed changes absurdly interactive: https://cerebrascoder.com/
I wonder if you didn't try cursor's Composer tab, especially set to Agent?
I didn't care that much for cursor when I was just using Chat but once I switched to Composer I was very happy, and my experience is in total disagreement that it's not so good for smaller projects.
They also must have a good prompt for diff-based completions, I don't know how hard it is to extract that.
> The biggest problem with all current codegen systems is the speed of generation
I don't see this complained about nearly as much as I'd expect. Groq has been out for over a year, I'm surprised OpenAI not acquired them and figured out how to 10x to 20x their speed on gpt4.
Yeah I don't agree. I'm building a product in the space, and the number one problem is correctness, not latency.
People are very happy to sit there for minutes if the correctness is high and the quality is high. It's still 100x or 1000x faster than finding 3rd party developers to work for you.
I wish the models were getting better but recently they've felt very stuck and this is it, so agent architectures will be the answer in the short term. That's what's working for us at srcbook rn.
I think the logic behind faster inference is that the LLM is unlikely to get it right the first time regardless of its intelligence simply due to the inherent ambiguity of human language. The faster it spits out a semi-functional bit of code the faster the iteration loop and the faster the user gets what they want.
Plus if you’re dealing with things like syntax errors, a really really fast llm + interpreter could report and fix the error in less than a minute with no user input.
I'm interested in what stopped you from finishing diffs and diff based editing. I built an AI software engineering assistant at my last company and we got decent results with Aider's method (and prompts, and hidden conversation starter etc). I did have to have a fallback to raw output, and a way to ask it to try again. But for the most part it worked well and unlocked editing large files (and quickly).
Excellent question! We just didn't have the resources at the time on our small team to invest in getting it to be good enough to be default on. We had to move on to other more core platform features.
Though I'm really eager to get back to it. When using Windsurf last week, I was impressed by their diffs on Sonnet. Seems like they work well. I would love to view their system prompt!
I hope that when we have time to resume work on this (maybe in Feb) that we'll be able to get it done. But then again, maybe just patience (and more fast-following) is the right strategy, given how fast things are moving...
An interesting alternative to diffs appears to be straightforward find and replace.
Claude Artifacts uses that: they have a tool where the LLM can say "replace this exact text with this" to update an Artifat without having to output the whole thing again.
What we found was that error handling on the client side was also very important. There's a bunch of that in Aider too for inspiration. Fuzzy search, indent fixing, that kind of stuff.
And also just to clarify, aider landed on search/replace blocks for gpt-4o and claude rather than actual diffs. We followed suit. And then we showed those in a diff UI client side
Readit News
To me, that's the most important consideration here. Are you targeting professional devs who are enhancing their current workflows iteratively with these improvements? Or re-thinking from the ground up, obfuscating most of what we've learned to date?
Maybe we need to trudge through all of these weeds until software creation hits its final, elegant form where "Anyone Can Code".
Maybe the old Gusteau quote is actually fitting here:
"You must be imaginative, strong-hearted. You must try things that may not work, and you must not let anyone define your limits because of where you come from. Your only limit is your soul. What I say is true - anyone can ̶c̶o̶o̶k̶ code... but only the fearless can be great."
Simplifying to the point a grandma could make an app isn't gonna make any grandma WANT to make apps. And that's fine, there's no issue, we don't have to make more people code and those who want, will, even if all we had was assembly and a light board...
Which I think is the spirit of your quote basically.
I do agree with your second paragraph and it’s more that you DON’T want to cook versus you being unable to cook.
- use the hoogle cli to search for the right types and functions
- include a comprehensive test suite
- run a build after every code change
- run tests after every successful build
GHC + a Claude-based agent is a thing to behold.
Deleted Comment
I've met so many engineers who have said exactly this. There are clearly some group of people obsessed with Cursor, but it's interesting to me how alien they seem to the majority of people using ai codegen right now.
"code" used to fire VS Code.
My rationale is: what else do they think they can get away with in my system?
VS Code Copilot Chat with #codebase in prompt has the edit mode which behaves similar to Cursor. Even more so with o1-preview selected.
Our first had a nice discussion on HN: https://blog.val.town/blog/codegen/
The other posts in the series:
- https://blog.val.town/blog/townie/
- https://blog.val.town/blog/building-a-code-writing-robot/
I think it is the short, punchy sections with plenty of visuals and the fact that you are telling a story the whole way through, which has a natural flow, each experiment you describe, leading to the next.
https://lmarena.ai/?leaderboard
Worth checking out their Cerebras-powered demo too - LLMs at 2000 tokens/second make applying proposed changes absurdly interactive: https://cerebrascoder.com/
I didn't care that much for cursor when I was just using Chat but once I switched to Composer I was very happy, and my experience is in total disagreement that it's not so good for smaller projects.
They also must have a good prompt for diff-based completions, I don't know how hard it is to extract that.
Yes, I wonder if reilly3000 will swing by with a leaked system prompt from them too
I don't see this complained about nearly as much as I'd expect. Groq has been out for over a year, I'm surprised OpenAI not acquired them and figured out how to 10x to 20x their speed on gpt4.
People are very happy to sit there for minutes if the correctness is high and the quality is high. It's still 100x or 1000x faster than finding 3rd party developers to work for you.
I wish the models were getting better but recently they've felt very stuck and this is it, so agent architectures will be the answer in the short term. That's what's working for us at srcbook rn.
Plus if you’re dealing with things like syntax errors, a really really fast llm + interpreter could report and fix the error in less than a minute with no user input.
Though I'm really eager to get back to it. When using Windsurf last week, I was impressed by their diffs on Sonnet. Seems like they work well. I would love to view their system prompt!
I hope that when we have time to resume work on this (maybe in Feb) that we'll be able to get it done. But then again, maybe just patience (and more fast-following) is the right strategy, given how fast things are moving...
https://www.reddit.com/r/LocalLLaMA/comments/1h7sjyt/windsur...
Claude Artifacts uses that: they have a tool where the LLM can say "replace this exact text with this" to update an Artifat without having to output the whole thing again.
ChatGPT's new Canvas feature apparently does a more sophisticated version of that using regular expressions as opposed to simple text matching: https://twitter.com/sh_reya/status/1875227816993943823
And also just to clarify, aider landed on search/replace blocks for gpt-4o and claude rather than actual diffs. We followed suit. And then we showed those in a diff UI client side