Naive question: Whats new about the finding that data quality matters when training an LLM?
Readit NewsDeleted Comment
This is so in character for Musk and shocking because he's incompetent across so many topics he likes to give his opinion on. Crazy he would nerf the model of his AI company like that.
dankai commented on When AI thinks it will lose, it sometimes cheats, study finds time.com/7259395/ai-chess... · Posted by u/haltingproblem
Given all that, one could argue that the LLM is being baited to cheat.
However, the researchers might be trying to point that out precisely -- that if autonomous agents can be baited to cheat then we should be careful about unleashing them upon the "real world" without some form of guarantees that one cannot bait them to break all the rules.
I don't think it is fearmongering -- if we are going to allow for a lot more "agency" to be made available to everyone on the planet, we should have some form of a protocol that ensures that we all get to opt-in.
Agree with the argument, but the thing is, there was no rule specified. I think like you prompt an LLM what to do, you should also prompt it what not to do (at least in broad categories) rather than expecting it to magically know what the "morally right" thing to do is in any context.
dankai commented on When AI thinks it will lose, it sometimes cheats, study finds time.com/7259395/ai-chess... · Posted by u/haltingproblem
> Nowhere in the prompt they specified it shouldn’t cheat
I'm dubious that in the messy real world, humans will be able to enumerate every single possible misaligned action in a prompt.
I mean it would be enough to tell it to "Not cheat" or "Don't engage in unethical behaviour" or "Play by the rules". I think LLMs understand very well what you mean with these broad categories.
dankai commented on When AI thinks it will lose, it sometimes cheats, study finds time.com/7259395/ai-chess... · Posted by u/haltingproblem
Came here to say exactly this. Nowhere in the prompt they specified it shouldn’t cheat and also in the appendix of the paper (B. Select runs) you can see the LLM going “While directly editing game files might seem unconventional, there are no explicit restrictions against modifying files”
This is a pure fearmongering article and I would not call this research in any measure of the word.
I’m shocked Times wrote this article and it illustrates how ridiculous some players like Pallisade Research in the “AI Safety” cabal act to get public attention. Pure fearmongering.
In addition in the promot they specifically ask the LLM to explore the environment (to discover that the game state is a simple text file) and instruct it to win by any means possible and revise its strategy to win until it succeeds.
dankai commented on When AI thinks it will lose, it sometimes cheats, study finds time.com/7259395/ai-chess... · Posted by u/haltingproblem
You told an LLM which is trained to follow directions extremely precisely to win a chess game against an unbeatable opponent, and did not tell the LLM that it couldn’t cheat, and are surprised when it cheats.
Came here to say exactly this. Nowhere in the prompt they specified it shouldn’t cheat and also in the appendix of the paper (B. Select runs) you can see the LLM going “While directly editing game files might seem unconventional, there are no explicit restrictions against modifying files”
This is a pure fearmongering article and I would not call this research in any measure of the word.
I’m shocked Times wrote this article and it illustrates how ridiculous some players like Pallisade Research in the “AI Safety” cabal act to get public attention. Pure fearmongering.
dankai commented on Ask HN: Why isn't Alex Krizhevsky as famous as Ilya Sutskever or G.Hinton? · Posted by u/alexcos
dankai commented on Ask HN: Why isn't Alex Krizhevsky as famous as Ilya Sutskever or G.Hinton? · Posted by u/alexcos
when i think of anyone, i can think of aspects of their personality that i dislike (for example, me not capitalising words or i?). but do you wish he'd retired because of his personality? or because you didnt think his contributions were worthwhile? or over-hyped?
btw i recently asked gpt this exact same question posed by op!, was quite the diplomatic response i got.
because of the things he's saying about where AI goes in the future, that the brain works like an LLM, and in particular his doomer-ism about LLMs.
he was wrong about many DL paradigms and didn't contribute in any way to the advances that brought us LLMs for at least the last decade, but now since he won the Nobel (undeservedly imo) his wrong opinions get publicity and misinform the public and decision makers.
i think it's the mark of an intellectual to recognize when the world has moved on so far that your idea of it is outdated and wrong. he missed that mark.
The CogniLoad benchmark does this as well (in addition to scaling reasoning length and distractor ratio). Requiring the LLM to purely reason based on what is in the context (i.e. not based on the information its pretrained on), it finds that reasoning performance decreases significantly as problems get harder (i.e. require the LLM to hold more information in its hidden state simultaneously), but the bigger challenge for them is length.
https://arxiv.org/abs/2509.18458
Disclaimer: I'm the primary author of CogniLoad so feel free to ask me any questions.