Readit NewsBut there are also the third kind: who like to design the systems and let them be built by someone else..
My experience is that people who weren't very good at writing software are the ones now "most excited" to "create" with a LLM.
LLMs goof up, hallucinate, make many mistakes - especially in design or architecting phase. That's where the experience truly shines.
Plus, it let's you integrate things that you aren't good at (UI for me).
I have a game written in XNA
100% of the code is there, including all the physics that I hand-wrote.
All the assets are there.
I tried to get Gemini and Claude to do it numerous times, always with utter failure of epic proportions with anything that's actually detailed. 1 - my transition from the lobby screen into gameplay? 0% replicated on all attempts 2 - the actual physics in gameplay? 0% replicated none of it works 3 - the lobby screen itself? non-functional
Okay so what did it even do? Well it put together sort of a boilerplate main menu and barebones options with weird looking text that isn't what I provided (given that I provided a font file), a lobby that I had to manually adjust numerous times before it could get into gameplay, and then nonfunctional gameplay that only handles directional movement and nothing else with sort of half-working fish traveling behavior.
I've tried this a dozen times since 2023 with AI and as late as late last year.
ALL of the source code is there every single thing that could be translated to be a functional game in another language is there. It NEVER once works or even comes remotely close.
The entire codebase is about 20,000 lines, with maybe 3,000 of it being really important stuff.
So yeah I don't really think AI is "really good" at anything complex. I haven't really been proven wrong in my 4 years of using it now.
But I do feel this is a solvable problem long term.
Are you generating revenue or, otherwise, what productivity are you measuring?
Without generating revenue (which to be clear is a very good proxy to measure impact) everyone can be indeed very prolific in their hobbies. But labor market is about making money for a living and unless you can directly impact your day-to-day needs from your work, it can't be called productive.
At my previous employer, I was generating $2.5million per year (revenue per employee). I didn't ship a single line of code. All the time was spent trying to convince various stake holders.
Now, I have already built a couple of apps that help me better manage my tech news (keeps me sane) plus I am writing a blog that generates $0. It's only been a month.
If you measure the immediate dollar value, you are right. But in life, pay-offs are not always realized immediately. Just my opinion anyway.
Here is my take on AI's impact on productivity:
First let's review what are LLMs objectively good at: 1. Writing boiler plate code 2. Translating between two different coding languages (migration) 3. Learning new things: Summarizing knowledge, explaining concepts 4. Documentation, menial tasks
At a big tech product company #1 #2 #3 are not as frequent as one would think - most of the time is spent in meetings and meetings about meetings. Things move slowly - it's designed to be like that. Majority devs are working on integrating systems - whatever their manager sold to their manager and so on. The only time AI really helped me at my job was when I did a one-week hackathon. Outside of that, integrations of AI felt like more work rather than less - without much productivity boost.
Outside, it has proven to be a real productivity boost for me. It checks all the four boxes. Plus, I don't have to worry about legal, integrations, production bugs (eventually those will come).
So, depends who you are asking -- it is a huge game changer (or not).
IMO, the writer is overzealous with their comments on LLMs. As a coder, it feels like an outsider trying out a product that was amazed me over and over so many times.
> They aren’t perfect, but the kind of analysis the program is able to do is past the point where technology looks like magic.
But as you use this product over a long period of time, there are many obvious gaps - hallucinations / repeated tool calls / out of context outputs / etc.
To me, refine.ink sounds like a company that has built heavy tooling around some super high context window LLMs and then some very good prompts. Their claim is to compare it against any good off-the-shelf LLM with any prompt. But when you are spending bunch of money to build a whole ecosystem around LLMs, it's obvious that it's not going to beat their output.
I won't be surprised if the next version of an LLM within the next few months completely outperforms their output -- that's usually the case with all the coding tools and scaffoldings. They are rendered useless by a superior LLM.Nevertheless, the initiative looks cool!
Two of my feature requests: 1. Allow cmd+f search on the whole app - I wanted to search your post on the app but I couldn't 2. A browser button to open the current page on an external browser.
Side note: I am trying to minimize my HN time via getting push notifications for relevant HN posts, and that's how I discovered your post. Would it be cool if one could write custom agents on top of an app? Maybe?
Two of my feature requests: 1. Allow cmd+f search on the whole app - I wanted to search your post on the app but I couldn't 2. A browser button to open the current page on an external browser.
Side note: I am trying to minimize my HN time via getting push notifications for relevant HN posts, and that's how I discovered your post. Would it be cool if one could write custom agents on top of an app? Maybe?
- one group loves to work independently and gets you the results, they are fast and they figure things out
- second group needs direction, they can be creative in their space but check-ins and course corrections are needed.
AI feels like group1 but it's actually group2. In essence, it doesn't fully fit in either group. I am still figuring out this third group.