Readit News logoReadit News
yuedongze commented on AI should only run as fast as we can catch up   higashi.blog/2025/12/07/a... · Posted by u/yuedongze
kristjank · 6 days ago
This feeling of verification >> generation anxiety bears a resemblance to that moment when you're learning a foreign language, you speak a well-prepared sentence, and your correspondent says something back, of which you only understand about a third.

In like fashion, when I start thinking of a programming statement (as a bad/rookie programmer) and an assistant completes my train of thought (as is default behaviour in VS Code for example), I get that same feeling that I did not grasp half the stuff I should've, but nevertheless I hit Ctrl-Return because it looks about right to me.

yuedongze · 6 days ago
> because it looks about right to me

this is something one can look in further. it is really probabilistic checkable proofs underneath, and we are naturally looking for places where it needs to look right, and use that as a basis of assuming the work is done right.

yuedongze commented on AI should only run as fast as we can catch up   higashi.blog/2025/12/07/a... · Posted by u/yuedongze
yuedongze · 6 days ago
It's nice to see a wide array of discussions under this! Glad that I didn't give up on this thought and end up writing it down.

I want to stress that the main point of my article is not really about AI coding, it's about letting AI perform any arbitrary tasks reliably. Coding is an interesting one because it seems like it's a place where we can exploit structure and abstraction and approaches (like TDD) to make verification simpler - it's like spot-checking in places with a very low soundness error.

I'm encouraging people to look for tasks other than coding to see if we can find similar patterns. The more we can find these cost asymmetry (easier to verify than doing), the more we can harness AI's real potential.

yuedongze commented on AI should only run as fast as we can catch up   higashi.blog/2025/12/07/a... · Posted by u/yuedongze
CuriouslyC · 6 days ago
It's architecture dependent. A fairly functional modular monolith with good documentation can be accessible to LLMs at the million line scale, but a coupled monolith or poorly instrumented microservices can drive agents into the ground at 100k.
yuedongze · 6 days ago
I think it's definitely an interesting subject for Verification Engineering. the easier to task AI to do work more precisely, the easier we can check their work.
yuedongze commented on AI should only run as fast as we can catch up   higashi.blog/2025/12/07/a... · Posted by u/yuedongze
jascha_eng · 6 days ago
Verification is key, and the issue is that almost all AI generated code looks plausible so just reading the code is usually not enough. You need to build extremely good testing systems and actually run through the scenarios that you want to ensure work to be confident in the results. This can be preview deployments or other AI generated end to end tests that produce video output that you can watch or just a very good test suite with guard rails.

Without such automation and guard rails, AI generated code eventually becomes a burden on your team because you simply can't manually verify every scenario.

yuedongze · 6 days ago
indeed, i see verification debt outweighing tradition tech debt very very soon...
yuedongze commented on AI should only run as fast as we can catch up   higashi.blog/2025/12/07/a... · Posted by u/yuedongze
drlobster · 6 days ago
I think you underestimating how good these image generators are at the moment.
yuedongze · 6 days ago
oh i mean the other direction! checking if a generated image is "good" that no one will tell something is off and it look naturally, rather than checking if they are fake.
yuedongze commented on AI should only run as fast as we can catch up   higashi.blog/2025/12/07/a... · Posted by u/yuedongze
gradus_ad · 6 days ago
The proliferation of nondeterministically generated code is here to stay. Part of our response must be more dynamic, more comprehensive and more realistic workload simulation and testing frameworks.
yuedongze · 6 days ago
i've seen a lot of startups that use AI to QA human work. how about the idea of use humans to QA AI work? a lot of interesting things might follow
yuedongze commented on AI should only run as fast as we can catch up   higashi.blog/2025/12/07/a... · Posted by u/yuedongze
rogerkirkness · 6 days ago
Appealing, but this is coming from someone smart/thoughtful. No offence to 'rest of world', but I think that most people have felt this way for years. And realistically in a year, there won't be any people who can keep up.
yuedongze · 6 days ago
im hoping this can introduce a framework to help people visualize the problem and figure out a way to close that gap. image generation is something every one can verify, but code generation is perhaps not. but if we can make verifying code as effortless as verifying images (not saying it's possible), then our productivity can enter the next level...

u/yuedongze

KarmaCake day338January 31, 2018View Original