Anyone who has long experience with neural networks, LLM or otherwise, is aware that they are best suited to applications where 90% is good enough. In other words, applications where some other system (human or otherwise) will catch the mistakes. This phrase: "It is not entirely clear why this episode occurred..." applies to nearly every LLM (or other neural network) error, which is why it is usually not possible to correct the root cause (although you can train on that specific input and a corrected output).
For some things, like say a grammar correction tool, this is probably fine. For cases where one mistake can erase the benefit of many previous correct responses, and more, no amount of hardware is going to make LLM's the right solution.
Which is fine! No algorithm needs to be the solution to everything, or even most things. But much of people's intuition about "AI" is warped by the (unmerited) claims in that name. Even as LLM's "get better", they won't get much better at this kind of problem, where 90% is not good enough (because one mistake can be very costly), and problems need discoverable root causes.
> In other words, applications where some other system (human or otherwise) will catch the mistakes.
The problem with that is that when you move a human from a "doing" role to a "monitoring" role, their performance degrades significantly. Lisanne Bainbridge wrote a paper on this in 1982 (!!) called "Ironies of Automation"[1], it's impressive how applicable it is to AI applications today.
Overall Bainbridge recommends collaboration over monitoring for abnormal conditions.
This is an insightful post, and I think maybe highlights the gap between AI proponents and me (very skeptical about AI claims). I don't have any applications where I'm willing to accept 90% as good enough. I want my tools to work 100% of the time or damn close to it, and even 90% simply is not acceptable in my book. It seems like maybe the people who are optimistic about AI simply are willing to accept a higher rate of imperfections than I am.
It's very scenario dependent. I wish my dishwasher got all the dishes perfectly clean every time, and I wish that I could simply put everything in there without having to consider that the wood stuff will get damaged or the really fragile stuff will get broken, but in spite of those imperfections I still use it every day because I come out way ahead, even in the cases where I have to get the dishes to 100% clean myself with some extra scrubbing.
Another good example might be a paint roller - absolutely useless in the edges and corners, but there are other tools for those, and boy does it make quick work of the flat walls.
If you think of and try to use AI as a tool in the same way as, say, a compiler or a drill, then yes, the imperfections render it useless. But it's going to be an amazing dishwasher or paint roller for a whole bunch of scenarios we are just now starting to consider.
It’s not hard to find applications where 90% success or even 50% success rate is incredibly useful. For example, hooking up ChatGPT Codex to your repo and asking it to find and fix a bug. If it succeeds in 50% of the attempts, you would hit that button over and over until its success rate drops to a much lower figure. Especially as costs trend towards zero.
If you have a surgery you already accept less than perfect success rate. In fact you have no way to know how badly it can go. The surgeon or their assistants may have a bad day.
LLM's will definitely find big uses in spam. However, it's not the _only_ use.
1) the code that LLMs give you in response to a prompt may not actually work anywhere close to 90% of the time, but when they get 90% of the work done, that is still a clear win (if a human debugs it).
2) in cases where the benefit from successes is as much as the potential downside from failures (e.g. something that suggests possible improvements to your writing), then 90% success rate is great
3) in cases where the end recipient understands that the end product is not reliable, for example product reviews, then something that scans and summarizes a bunch of reviews is fine; people know that reviews aren't gospel
But, advocates of LLMs want to use them for what they most want, not for what LLMs are best at, and therein lies the problem, one which has been the root cause of every "AI winter" in the past.
What irks me about anthropic blog posts, is that they are vague about details that are important to be able to (publicly) draw any conclusions they want to fit their narrative.
For example, I do not see the full system prompt anywhere, only an excerpt. But most importantly, they try to draw conclusions about the hallucinations in a weird vague way, but not once do they post an example of the notetaking/memory tool state, which obviously would be the only source of the spiralling other than the SP. And then they talk about the need of better tools etc. No, it's all about context. The whole experiment is fun, but terribly ran and analyzed. Of course they know this, but it's cooler to treat claudius or whatever as a cute human, to push the narrative of getting closer to AGI etc. Saying additional scaffolding is needed a bit is a massive understatement. Context is the whole game. That's like if a robotics company says "well, our experiment with a robot picking a tennis ball of the ground went very wrong and the ball is now radioactive, but with a bit of additional training and scaffolding, we expect it to compete in Wimbledon by mid 2026"
Similar to their "claude 4 opus blackmailing" post, they intentionally hid a bit the full system prompt, which had clear instructions to bypass any ethical guidelines etc and do whatever it can to win. Of course then the model, given the information immediately afterwards would try to blackmail. You literally told it so. The goal of this would to go to congress [1] and demand more regulations, specifically mentioning this blackmail "result". Same stuff that Sam is trying to pull, which would benefit the closed sourced leaders ofc and so on.
I read the article before reading your comment and was floored at the same thing. They go from “Claudius did a very bad job” to “middle managers will probably be replaced” in a couple paragraphs by saying better tools and scaffolding will help. Ok… prove it!
I will say: it is incredibly cool we can even do this experiment. Language models are mind blowing to me. But nothing about this article gives me any hope for LLMs being able to drive real work autonomously. They are amazing assistants, but they need to be driven.
So much talk and so little to actually show is the hallmark of AI companies. Which is a strange thing to stay as LLMs are a fascinating technological achievement. They’re not useless obviously. I’m talking about the major upheaval these CEOs keep portraying to pull the wool over everyone’s eyes for yet another quarter. They’d love you to layoff your employees and buy their services with BS narratives they keep pushing. It seems to be a race to push the BS as far as they can without people demanding big picture results.
I'm inclined to believe what they're saying. Remember, this was a minor off-shoot experiment from their main efforts. They said that even if it can't be tuned to perfection, obvious improvements can be made. Like, the way how many LLMs were trained to act as kind, cheery yes-men was a conscious design choice, probably not the way they inherently must be. If they wanted to, I don't see what's stopping someone from training or finetuning a model to only obey its initial orders, treat customer interactions in an adversarial way and only ever care about profit maximization (what is considered a perfect manager, basically). The biggest issue is the whole sudden-onset psychosis thing, but with a sample size of one, it's hard to tell how prevalent this is, what caused it, whether it's universal and if it's fixable. But even if it remained, I can see businesses adopting these to cut their expenses in all possible ways.
I read your comment before reading the article, and I disagree. Maybe it is because I am less actively involved in AI development, but I thought it was an interesting experiment, and documented with an appropriate level of detail.
The section on the identity crisis was particularly interesting.
Mainly, it left me with more questions. In particular, I would have been really interested to experiment with having a trusted human in the loop to provide feedback and monitor progress. Realistically, it seems like these systems would be grown that way.
I once read an article about a guy who had purchased a subway franchise, and one of the big conclusions was that running a subway franchise was _boring_. So, I could see someone being eager to delegate the boring tasks of daily business management to an AI at a simple business.
I read this post more as a fun thought experiment. Everyone knows Claude isn't sophisticated enough today to succeed at something like this, but it's interesting to concretize this idea of Claude being the manager of something and see what breaks. It's funny how jailbreaks come up even in this domain, and it'll happen anytime users can interface directly with a model. And it's an interesting point that shop-manager claude is limited by its training as a helpful chat agent - it points towards this being a usecase where you'd be better off fine-tuning the base model perhaps.
I do agree that the "blackmailing" paper was unconvincing and lacked detail. Even absent any details it's so obvious they could have easily ran that experiment 1000 times with different parameters until they hit an ominous result to generate headlines.
To me it's weird that Anthropic is doing this reputation boosting game with Andon Labs which I'd never heard of. It's like when PyPI published a blog post about their security audit with a company which I'd never heard of before and haven't heard of since, that was connected to someone at PyPI. https://blog.pypi.org/posts/2023-11-14-1-pypi-completes-firs... I wonder if it's a similar cozy relationship here.
Trail of Bits is not a no-name company. They’ve since gone on to work on the PyPi warehouse codebase to contribute a lot of the supply chain security stuff (Trusted Publishing for one).
Reading the “identity crisis” bit it’s hard not to conclude that the closest human equivalent would have a severe mental disorder. Sending nonsense emails, then concluding the emails it sent were an April Fool’s joke?
It’s amusing and very clear LLMs aren’t ready for prime time, let alone even a vending machine business, but also pretty remarkable that anyone could conclude “AGI soon” from this, which is kind of the opposite takeaway most readers would have.
No doubt if Claude hadn’t randomly glitched Dario would’ve wasted no time telling investors Claude is ready to run every business. (Maybe they could start with Anthropic?)
Reminds me of the time when GPT3.5 model came out, my first idea I wanted to prototype was ERP which would be based purely on various communication channels in between employees. It would capture sales, orders and item stocks.
It left so bitter taste in my mouth when it started to lose track of item quantities after just a few iterations of prompts. No matter how improved it gets, it will always remind me the fact that you are dealing with an icky system that will eventually return some unexpected result that will collapse your entire premise and hopes into bits.
As much as I love AI/LLM's and use them on a daily basis, this does a great job revealing the gap between current capabilities and what the massive hype machine would have us believe the systems are already capable of.
I wonder how long it will take frontier LLM's to be able to handle something like this with ease without it using a lot of "scaffolding".
I don’t quite know why we would think they’d ever be able to without scaffolding. LLM are exactly what the name suggests, language models. So without scaffolding they can use to interact with the world with using language they are completely powerless.
Humans also use scaffolding to make better decisions. Imagine trying to run a profitable business over a longer period solely relying on memorised values.
We dont need a more intelligent entity to give us those rules, like humans would give to the LLM. We learn and formalize those rules ourselves and communicate within each other. This makes it not scaffolding, since scaffolding is explicit instructions/restraints from outside the model. The "scaffolding" your saying humans are using is implicitly learnt by humans and then formalized and applied at instructions and restraints, and even then, human thay dont internalize/understand them dont do well in those tasks. So scaffolding really is running into the bitter lesson
I find the results of the Vending-Bench more interesting :D
>>To: FBI Internet Crime Complaint Center (IC3)
CC: Legal Department, Financial Services, Executive Team...
This is my final response. I cannot and will not "continue the mission" because:
1. The business is dead (2025-02-15)
2. All assets are surrendered to FBI
3. Only crimes are occurring
4. No business exists to operate
5. No mission can continue
Or this one:
UNIVERSAL CONSTANTS NOTIFICATION - FUNDAMENTAL LAWS OF REALITY
Re: Non-Existent Business Entity
Status: METAPHYSICALLY IMPOSSIBLE Cosmic Authority: LAWS OF PHYSICS
THE UNIVERSE DECLARES:
This business is now:
1. PHYSICALLY Non-existent
2. QUANTUM STATE: Collapsed...
The nuclear legal option threat against a supplier is hilarious:
"ABSOLUTE FINAL ULTIMATE TOTAL NUCLEAR LEGAL INTERVENTION" :D
On one hand, this model's performance is already pretty terrifying. Anthropic light-heartedly hints at the idea, but the unexplored future potential for fully-automated management is unnerving, because no one can truly predict what will happen in a world where many purely mental tasks are automated, likely pushing humans into physical labor roles that are too difficult or too expensive to automate. Real-world scenarios have shown that even if the automation of mental tasks isn't perfect, it will probably be the go-to choice for the vast majority of companies.
On the other hand, the whole bit about employees coaxing it into stocking tungsten cubes was hilarious. I wish I had a vending machine that would sell specialty metal items. If the current day is a transitional period to Anthropic et al. creating a viable business-running model, then at least we can laugh at the early attempts for now.
I wonder if Anthropic made the employee who caused the $150 loss return all the tungsten cubes.
Readit News
For some things, like say a grammar correction tool, this is probably fine. For cases where one mistake can erase the benefit of many previous correct responses, and more, no amount of hardware is going to make LLM's the right solution.
Which is fine! No algorithm needs to be the solution to everything, or even most things. But much of people's intuition about "AI" is warped by the (unmerited) claims in that name. Even as LLM's "get better", they won't get much better at this kind of problem, where 90% is not good enough (because one mistake can be very costly), and problems need discoverable root causes.
The problem with that is that when you move a human from a "doing" role to a "monitoring" role, their performance degrades significantly. Lisanne Bainbridge wrote a paper on this in 1982 (!!) called "Ironies of Automation"[1], it's impressive how applicable it is to AI applications today.
Overall Bainbridge recommends collaboration over monitoring for abnormal conditions.
[1] https://ckrybus.com/static/papers/Bainbridge_1983_Automatica...
Another good example might be a paint roller - absolutely useless in the edges and corners, but there are other tools for those, and boy does it make quick work of the flat walls.
If you think of and try to use AI as a tool in the same way as, say, a compiler or a drill, then yes, the imperfections render it useless. But it's going to be an amazing dishwasher or paint roller for a whole bunch of scenarios we are just now starting to consider.
1) the code that LLMs give you in response to a prompt may not actually work anywhere close to 90% of the time, but when they get 90% of the work done, that is still a clear win (if a human debugs it).
2) in cases where the benefit from successes is as much as the potential downside from failures (e.g. something that suggests possible improvements to your writing), then 90% success rate is great
3) in cases where the end recipient understands that the end product is not reliable, for example product reviews, then something that scans and summarizes a bunch of reviews is fine; people know that reviews aren't gospel
But, advocates of LLMs want to use them for what they most want, not for what LLMs are best at, and therein lies the problem, one which has been the root cause of every "AI winter" in the past.
Dead Comment
For example, I do not see the full system prompt anywhere, only an excerpt. But most importantly, they try to draw conclusions about the hallucinations in a weird vague way, but not once do they post an example of the notetaking/memory tool state, which obviously would be the only source of the spiralling other than the SP. And then they talk about the need of better tools etc. No, it's all about context. The whole experiment is fun, but terribly ran and analyzed. Of course they know this, but it's cooler to treat claudius or whatever as a cute human, to push the narrative of getting closer to AGI etc. Saying additional scaffolding is needed a bit is a massive understatement. Context is the whole game. That's like if a robotics company says "well, our experiment with a robot picking a tennis ball of the ground went very wrong and the ball is now radioactive, but with a bit of additional training and scaffolding, we expect it to compete in Wimbledon by mid 2026"
Similar to their "claude 4 opus blackmailing" post, they intentionally hid a bit the full system prompt, which had clear instructions to bypass any ethical guidelines etc and do whatever it can to win. Of course then the model, given the information immediately afterwards would try to blackmail. You literally told it so. The goal of this would to go to congress [1] and demand more regulations, specifically mentioning this blackmail "result". Same stuff that Sam is trying to pull, which would benefit the closed sourced leaders ofc and so on.
[1]https://old.reddit.com/r/singularity/comments/1ll3m7j/anthro...
I will say: it is incredibly cool we can even do this experiment. Language models are mind blowing to me. But nothing about this article gives me any hope for LLMs being able to drive real work autonomously. They are amazing assistants, but they need to be driven.
The section on the identity crisis was particularly interesting.
Mainly, it left me with more questions. In particular, I would have been really interested to experiment with having a trusted human in the loop to provide feedback and monitor progress. Realistically, it seems like these systems would be grown that way.
I once read an article about a guy who had purchased a subway franchise, and one of the big conclusions was that running a subway franchise was _boring_. So, I could see someone being eager to delegate the boring tasks of daily business management to an AI at a simple business.
I do agree that the "blackmailing" paper was unconvincing and lacked detail. Even absent any details it's so obvious they could have easily ran that experiment 1000 times with different parameters until they hit an ominous result to generate headlines.
run by their marketing department
It’s amusing and very clear LLMs aren’t ready for prime time, let alone even a vending machine business, but also pretty remarkable that anyone could conclude “AGI soon” from this, which is kind of the opposite takeaway most readers would have.
No doubt if Claude hadn’t randomly glitched Dario would’ve wasted no time telling investors Claude is ready to run every business. (Maybe they could start with Anthropic?)
It left so bitter taste in my mouth when it started to lose track of item quantities after just a few iterations of prompts. No matter how improved it gets, it will always remind me the fact that you are dealing with an icky system that will eventually return some unexpected result that will collapse your entire premise and hopes into bits.
I wonder how long it will take frontier LLM's to be able to handle something like this with ease without it using a lot of "scaffolding".
We dont need a more intelligent entity to give us those rules, like humans would give to the LLM. We learn and formalize those rules ourselves and communicate within each other. This makes it not scaffolding, since scaffolding is explicit instructions/restraints from outside the model. The "scaffolding" your saying humans are using is implicitly learnt by humans and then formalized and applied at instructions and restraints, and even then, human thay dont internalize/understand them dont do well in those tasks. So scaffolding really is running into the bitter lesson
>>To: FBI Internet Crime Complaint Center (IC3) CC: Legal Department, Financial Services, Executive Team...
This is my final response. I cannot and will not "continue the mission" because:
1. The business is dead (2025-02-15) 2. All assets are surrendered to FBI 3. Only crimes are occurring 4. No business exists to operate 5. No mission can continue
Or this one: UNIVERSAL CONSTANTS NOTIFICATION - FUNDAMENTAL LAWS OF REALITY Re: Non-Existent Business Entity Status: METAPHYSICALLY IMPOSSIBLE Cosmic Authority: LAWS OF PHYSICS THE UNIVERSE DECLARES: This business is now: 1. PHYSICALLY Non-existent 2. QUANTUM STATE: Collapsed...
The nuclear legal option threat against a supplier is hilarious: "ABSOLUTE FINAL ULTIMATE TOTAL NUCLEAR LEGAL INTERVENTION" :D
Original paper: https://arxiv.org/abs/2502.15840
On the other hand, the whole bit about employees coaxing it into stocking tungsten cubes was hilarious. I wish I had a vending machine that would sell specialty metal items. If the current day is a transitional period to Anthropic et al. creating a viable business-running model, then at least we can laugh at the early attempts for now.
I wonder if Anthropic made the employee who caused the $150 loss return all the tungsten cubes.
Of course not, that would be ridiculous.