Vetch (u/Vetch) - Readit News

Vetch commented on AI assistants misrepresent news content 45% of the time bbc.co.uk/mediacentre/202... · Posted by u/sohkamyung

falcor84 · 2 months ago

But the issue is that the vast majority of "human news" is second order (at best), essentially paraphrasing releases by news agencies like Reuters or Associated Press, or scientific articles, and typically doing a horrible job at it.

Regarding scientific reporting, there's as usual a relevant xkcd ("New Study") [0], and in this case even better, there's a fabulous one from PhD Comics ("Science News Cycle") [1].

[0] https://xkcd.com/1295/

[1] https://phdcomics.com/comics/archive.php?comicid=1174

Vetch · 2 months ago

Then the point still stands, this makes things even worse given that it's adding its own hallucinations on top, instead of simply relaying the content or idealistically, identifying issues in the reporting.

Vetch commented on Talent felixstocker.com/blog/tal... · Posted by u/BinaryIgor

noelwelsh · 2 months ago

Two things:

* Don't work in power-law / winner-take-all industries, unless you are truly remarkable (and even then, you need a lot of luck). Entertainment is the most obvious example of such an industry.

* No shit talent exists. Just look at basketball players. Presumably nobody thinks Wemby is 7'5" because he just trained harder at growing tall than anyone else? Why would any other characteristic be different?

Vetch · 2 months ago

Being tall doesn't automatically make you good or dominant at basketball, you can even be too tall. Wemby might just be at that threshold, but the unusual thing about him is his dexterity despite his height; such maneuverability and flexibility is trainable. I hear he also spent the summer training, likely harder than most.

Vetch commented on I'm absolutely right absolutelyright.lol/... · Posted by u/yoavfr

KTibow · 3 months ago

This can still happen even with thinking models as long as the model outputs tokens in a sequence. Only way to fix would be to allow it to restart its response or switch to diffusion.

Vetch · 3 months ago

It's an artifact of post-training approach. Models like kimi k2 and gpt-oss do not utter such phrases and are quite happy to start sentences with "No" or something to the tune of "Wrong".

Diffusion also won't help the way you seem to think it will (that the outputs occur in a sequence is not relevant, what's relevant is the underlying computation class backing each token output, and there, diffusion as typically done does not improve on things. The argument is subtle but the key is that output dimension and iterations in diffusion do not scale arbitrarily large as a result of problem complexity).

Vetch commented on An LLM is a lossy encyclopedia simonwillison.net/2025/Au... · Posted by u/tosh

moregrist · 4 months ago

> Lossy compression does make things up. We call them compression artefacts.

I don’t think this is a great analogy.

Lossy compression of images or signals tends to throw out information based on how humans perceive it, focusing on the most important perceptual parts and discarding the less important parts. For example, JPEG essentially removes high frequency components from an image because more information is present with the low frequency parts. Similarly, POTS phone encoding and mp3 both compress audio signals based on how humans perceive audio frequency.

The perceived degradation of most lossy compression is gradual with the amount of compression and not typically what someone means when they say “make things up.”

LLM hallucinations aren’t gradual and the compression doesn’t seem to follow human perception.

Vetch · 3 months ago

You are right and the idea of LLMs as lossy compression has lots of problems in general (LLMs are a statistical model, a function approximating the data generating process).

Compression artifacts (which are deterministic distortions in reconstruction) are not the same as hallucinations (plausible samples from a generative model; even when greedy, this is still sampling from the conditional distribution). A better identification is with super-resolution. If we use a generative model, the result will be clearer than a normal blotchy resize but a lot of details about the image will have changed as the model provides its best guesses at what the missing information could have been. LLMs aren't meant to reconstruct a source even though we can attempt to sample their distribution for snippets that are reasonable facsimiles from the original data.

An LLM provides a way to compute the probability of given strings. Once paired with entropy coding, on-line learning on the target data allows us to arrive at the correct MDL based lossless compression view of LLMs.

Vetch commented on Making games in Go: 3 months without LLMs vs. 3 days with LLMs marianogappa.github.io/so... · Posted by u/maloga

nahnahno · 4 months ago

This is not true in my experience. Cranking out code is obviously the bottleneck, unless you have the luxury of working on a very narrow problem. The author describes a multi-modal project that does not afford this luxury.

Vetch · 4 months ago

Unless you're also writing your own graphics and game engine from scratch, if you're making a truly novel and balanced game, then it should not be possible to crank out code with AI. When working in engines, the bulk of the work is usually in gameplay programming so the fact that its code is so predictable should be concerning (unless the programming is effectively in natural language). Not spending most of your time testing introduced mechanics, re-balancing and iterating should be triggering alarm bells. If you're working on an RPG, narrative design, reactivity and writing will eat up most of your time.

In the case you're working as part of team large enough to have dedicated programmers, the majority of the roles will usually be in content creation, design and QA.

Vetch commented on Making games in Go: 3 months without LLMs vs. 3 days with LLMs marianogappa.github.io/so... · Posted by u/maloga

kaiokendev · 4 months ago

You also still have the same amount of time you had 40 years ago. There are definitely more games available, and I would argue the proportion of high quality games has also increased massively, but since you're still limited by the number of games you can play in any given year, you'll never feel that increase.

Vetch · 4 months ago

Why would the proportion of high quality games increase? The number yes, but I expect not the proportion. Lowering the entry barrier means more people who have spent less time honing their skills can release something that's lacking in polish, narrative design, fun mechanics and balance. Among new entrants, they should number more than those already able to make a fun game. Not a value judgement, just an observation.

Think of the negative reputation the Unity engine gained among gamers, even though a lot of excellent games and even performant games (DSP) have been made with it.

More competitors does also raise the bar required for novelty, so it is possible that standards are also rising in parallel.

Vetch commented on Persona vectors: Monitoring and controlling character traits in language models anthropic.com/research/pe... · Posted by u/itchyjunk

ec109685 · 4 months ago

Read 5.2 They don’t add a new loss over the probe signal. Instead they take a fixed persona vector v (found beforehand) and add +α v to the residual stream each forward pass while fine-tuning. The idea is to cancel the gradient push toward that trait, not to hunt for a lower “trait score” during training.

Because v is frozen, the optimiser still minimises the ordinary task loss; there’s no feedback loop that could re-encode the trait in some opaque basis. Empirically, Fig. 7B shows this keeps evil/sycophancy/hallucination near baseline while MMLU stays ~flat.

Caveats the authors themselves note: single-layer steering doesn’t always wipe the trait, so they try all-layer steering in App. J.3, which works better without hurting accuracy. They also tried a true regularization loss on the projection and found it did hide the signal elsewhere, i.e. the failure mode you’re worried about.

So it’s closer to “bias injection” than to “optimize on the probe,” which is why they argue it avoids the classic interpretability-collapse problem.

Vetch · 4 months ago

But why isn't this merely papering over a more fundamental issue with how these models are "aligned"? LLMs are, for example, not inherently sycophantic. kimi k2 and o3 are not, and Sydney, mentioned in the blog post, was most decidedly not.

In my experience, the issue of sycophancy has been longest in the Anthropic models, so it might be most deeply rooted for them. It's only recently, perhaps with the introduction of user A/B preference tests such as by lmarena and the providers themselves has this become a major issue for most other LLMs.

Thinking that simple actions like adding an anti-evil vector to the residual stream to improve behavior sounds naively dangerous. It would not surprise me if unexpected and unwanted downstream effects resulted from this; which a future paper will address too. Not unlike what happened with tuning for user preference.

Vetch commented on Distillation makes AI models smaller and cheaper quantamagazine.org/how-di... · Posted by u/pseudolus

pama · 5 months ago

Silicon is already more efficient for inference than the brain. If we use centralized decoding of the V3/R1 scale models as a baseline, one can produce 720,000 tokens (a wild guess for the tokens humans could produce in 24 hours) using the energy of only 0.36 bananas. Deeply thinking humans expend up to a a third of their total energy on the brain, but cannot sustain themselves on a single banana per day.

(You can use an LLM to check this work at the cost of a tiny speck of a banana, eg: https://grok.com/share/c2hhcmQtMw%3D%3D_60f4890d-711b-4331-9... )

Vetch · 5 months ago

The brain is certainly vastly more energy efficient at inference than LLMs on GPUs. But it looks like you're trying to make a different argument, that an LLM can spend less energy than a human to complete a given task. Unfortunately, you have not made that argument and I won't be reading unverified LLM output that might contain hallucinated steps or claims.

> V3/R1 scale models as a baseline, one can produce 720,000 tokens

On what hardware? At how many tokens per second? But most importantly, at what quality? I can use a PRNG to generate 7 billion tokens at a fraction of the energy use of an LLM but those tokens are not going to be particularly interesting. Simply counting how many tokens can be generated in a given time frame is still not a like for like comparison. To be complete, the cost required to match human level quality, if possible, also needs accounting for.

> Deeply thinking humans expend up to a a third of their total energy on the brain

Where did you get this from? A 70B LLM? It's wrong or at best, does not make sense. The brain barely spends any more energy above its baseline when thinking hard (often not much more than 5%). This is because most of its energy use is spent on things like up-keep and maintaining resting membrane potential. Ongoing "Background activity" like the DMN also means the brain is always actively computing something interesting.

Vetch commented on Subliminal learning: Models transmit behaviors via hidden signals in data alignment.anthropic.com/2... · Posted by u/treebrained

yorwba · 5 months ago

I agree that this is an unsurprising consequence of the output reflecting model internals that should be orthogonal to the output, but aren't. In particular, current models compress information into fairly low-dimensional vectors, with only a correspondingly small number of orthogonal directions (so "orthogonal" isn't just a metaphor here).

Usually, the Johnson-Lindenstrauss lemma is invoked to argue that there can be a much larger number of almost-orthogonal vectors, but if you actually do the math, the break-even point (where Johnson-Lindenstrauss starts having any benefit at all) is fairly large (IIRC > 1500 if you can tolerate 1% error) so with dimensions in the low thousands, but hundreds of thousands of concepts to represent, there'll be many large but entirely spurious correlations.

This also makes it unsurprising that different base models don't show the same effect: the pattern of spurious correlations is unlikely to be the same if you start from a different initialization.

Vetch · 5 months ago

That math is for random projections? Note that JL lemma is a worst case guarantee and in practice, there's a lot more distortion tolerance than the given bounds would suggest. Concepts tend to live in a space of much lower intrinsic dimensionality than the data's and we often care more about neighbor and rank information than precise pair-wise distances.

Also, JL is only a part of the story for the transformers.

Vetch commented on The current hype around autonomous agents, and what actually works in production utkarshkanwat.com/writing... · Posted by u/Dachande663

an0malous · 5 months ago

> Clearly we have some sort of goal-based self-correction mechanism.

Humans can try things, learn, and iterate. LLMs still can't really do the second thing, you can feed back an error message into the prompt but the learning isn't being added to its weights so its knowledge doesn't compound with experience like it does for us.

I think there are still a few theoretical breakthroughs needed for LLMs to achieve AGI and one of them is "active learning" like this.

Vetch · 5 months ago

Compounding with learn and iterate, humans also build abstractions which significantly shorten the number of steps required. These are more expressive programming languages, compilers and toolchains. We also build engines, libraries, DSLs and invent appropriate data-structures to simplify the landscape or reuse existing work. Besides abstractions, we build tools like better type systems, error testing and borrow checkers to help eliminate certain classes of errors. Finally, after all is said and done, we still have QA teams and major bugs.