Ask HN: If LLMs are so useful, why haven't we seen any spike in productivity?

Writing code is the easiest part of the process (relatively speaking). Figuring out the requirements, working with stakeholders to drive consensus, and understanding user needs is the bulk of the work.

LLMs will certainly lower the entry barriers for new programmers, and might also create a new solopreneur economy because of it. Now non-technical people with ideas can start prototyping and raise money, but would soon need engineers to grow the product.

globular-toast · a year ago

That doesn't completely explain it. There are tons of open source projects with countless feature requests just begging for someone to implement them. There's not really any research to do. Often the feature being requested is ridiculously simple but difficult to implement.

andyp-kw · a year ago

I think this is a good point to consider.

Let's imagine an inexperienced developer comes across a problem in an open source library, that has an existing issue raised in GitHub.

Are tools like Copilot and ChatGPT good enough to walk them through setting up the dev environment, fixing code and testing the fix. Maybe, but not without many prompts from the dev.

But how is that different from someone StackOverflowing their way through the problem.

dewey · a year ago

Often it's not about "just" adding the feature, but evaluating if it makes sense, how it _should_ be implemented and if it should be part of some different feature. Just hacking something in is usually not the complicated part.

gls2ro · a year ago

I think we need to consider that sometimes a feature is not implemented or a PR is not merged also because any code added needs to be maintained.

So even if there are a lot of feature requests that does not mean that the maintainer wants to just implement them in any fast way because that is code/feature that needs maintenance further down.

Technology spreads slowly. google docs is an instant 50x productivity increase for any legal process and yet a few years ago I saw an advocate's mind blown by a simple demonstration of simultaneous editing from two people in the same affidavit.

For him, the norm is still to redline a document on paper, and have his secretary add those changes to the original digital document and have that sent over to the opposing team for the same treatment.

I don't have strong opinions about LLMs' coding ability (though compared to the other comments so far I am more on the "LLMs are pretty good at creating software from natural language descriptions" side) but even assuming that LLMs can give programmers a 50x productivity increase, I'd assume it would take 10-50 years for industry and processes to evolve to take advantage of that increase.

eesmith · a year ago

My lawyer says his office runs on Microsoft 365. Is Google Docs really 50x over that? I don't even see how it's 50x over LibreOffice and a shared drive.

tocs3 · a year ago

I think the "50x" is a little bit of a random number.

If you are already writing good code it might be hard to get any great improvement. If you are a beginner without much training /experience it might not be hard to see orders of magnitude improvement.

It might take some time though. When I have spoken to non coding people they seem to look at me like I am talking about flying to the moon. If computers are ever considered general tools and the general public every moves more towards more DIY and small business there might be more of an uptake.

sixhobbits · a year ago

Sure if it's online and multiplayer it's similar. My comment is not google over Microsoft but shared/multiplayer docs for tracking changes and reaching agreements in minutes for something that used to take (and for many still does take) weeks or months of manual reconciliation of printed paper documents

proc0 · a year ago

leshokunin · a year ago

Kudos for raising an empirical point rather than looking at the aspirations of the tech. It's hard to have that kind of look.

Jury's still out. It will take time until we have enough post mortems to tell if it is doing the job and how it's affecting things.

I do agree that if it was so good, we'd see practical applications ib more meaningful ways than just anecdotal tricks or lots of low quality content.

Turns out the bottleneck of engineering isn't related to what goes on in the editor.

pera · a year ago

I agree with your comment to certain degree, specially in a commercial enterprise environment, but programming is still very time consuming and if a new invention made us faster it would be something noticeable, no?

talldayo · a year ago

Just speaking from my experience, here; when I sit down to code, actually typing out the logic of what I want is not what I spend my time doing. I research optimization options, I prep old code for new features and refactor cruft. I have an AI-enabled editor, but besides generating boilerplate the AI-based features are mostly useless. My job doesn't rely on endlessly generating buggy code, it depends on the existence of endless buggy code that needs correcting.

xenospn · a year ago

You don’t think bespoke emojis are a boost to productivity?

Tepix · a year ago

What statistics are you referring to when making these claims?

Github hosts only 20% public repositories. Perhaps open source developers are less likely to have Github Copilot paid out of their own pocket?

Why do you expect "an explosion of new projects" with perhaps 20% of increased productivity? What percentage of open source developers are using LLMs for increased productivity when working on open source? If it's merely 20%, we'd see a 4% increase, something that's hardly noticeable.

dave4420 · a year ago

My employer pays GitHub ~£10/month for me to use GitHub’s copilot. This is tiny compared to what they pay me.

It unlocks a small amount of extra productivity, but not that much. Yet still enough to be worth it.

My position is that they are useful but not massively useful, yet.

Mc91 · a year ago

LLMs have been getting better - they were all pretty poor for my programming purposes a year or so ago, recently Perplexity (even the non-Pro version) and GPT4 have been helpful, and 4o is even better. I have been posting Leetcode hard problems into 4o and getting sensible outputs, something I didn't even try previously. Sometimes I do have to have it go through a few iterations, and I give it various qualifications (like keep to such-and-such time and space complexity or better). My usual instruction is to make the class or function more and more compact while keeping to the same functionality and time/space complexity.

I got 4o to give me a 33 line, relatively simple and understandable bidirectional BFS Kotlin function for this Leetcode problem which Perplexity (non-Pro) and GPT4 could solve, but not as well as 4o - https://leetcode.com/problems/word-ladder

Of course, even though these are Leetcode hard level problems, they are well-defined and relatively self-contained. I work at a Fortune 100 company and 99% of the time I can pound out the CRUD I do in my sleep - the difficulties I encounter are distractions, the CI server having some problem, the ticket/story I am working out not being fully specified and the PM is MIA that day, all teams are working on the feature at the same time and I need to find out what feature flags to have set and which test headers have been agreed on, the PM has asked me to work on something but some of what he says does not make sense in context so I have to ask for clarification etc. Then there's the meta-game of knowing what to prioritize, with one important component being what will make my manager happy so I get a good yearly review, and what I need to prioritize may differ from what my PM says to prioritize, or even more complexly, what my manager says to prioritize, but doesn't really mean.

j0hnyl · a year ago

I actually think all of the things you listed are absolutely happening.

moffkalast · a year ago

> for instance, in the open source community

They're definitely wrong on that point, there's countless projects that exist that otherwise wouldn't have been started at all. Anecdotally I would never have put in the initial effort to set up a project that has 100+ stars now without the initial kick from early GPT-4 last year.

Lots of these new repos are also disproportionately in the LLM related space specifically since that's where people use them the most for code, so it's probably not as noticeable at large yet.