Readit News logoReadit News
CoopaTroopa · 16 days ago
"The parable of the earring was not about the dangers of using technology that wasn't Truly Part Of You, which would indeed have been the kind of dystopianism I dislike. It was about the dangers of becoming too powerful yourself."

https://web.archive.org/web/20121007235422/http://squid314.l...

AndrewDucker · 16 days ago
As I said in a comment on that post, 13 years ago: "any parable that's about being too powerful is almost necessarily also about technology, because it's technology that allows the average person to get that power"
ameliaquining · 16 days ago
True, but concerns about LLMs with anything like current capabilities are of the "Truly Part of You" flavor, not the "becoming too powerful" flavor.

Deleted Comment

kazinator · 16 days ago
But before stating that intent, he admits "well, that parable didn't work". The strongest interpretation of which is that the parable he wrote didn't succeed in being about "the dangers of becoming too powerful".

We have to read "was about" as "was (supposed to be) about".

What the parable ends up being about is any consistent interpretation well supported by the actual text of the parable!

In the parable, the Whispering Earring is a kind of character. It has autonomy and agency; a mind of its own, separate from that of the wearer. It generates ideas and suggests them to the wearer, eventually rendering most of their brain unnecessary. (The implication being that the individual, as a sentient being, has wasted away and has been effectively replaced by the host, as if possessed in the classical sense).

Someone who could be just as powerful in making all the right decisions guaranteed to make them happy, but using their own brain instead of taking suggestions from a whispering daemonic oracle, would not waste away and be replaced; their brain would have to be doing remarkable work and developing in the process rather than atrophying.

I suspect that it would actually be very difficult to repair the parable, while retaining the key element of the Whispering Earring as an autonomous entity, into being about "the dangers of becoming too powerful oneself". (Has the author tried?)

bananaflag · 16 days ago
Thanks! Even though I have the whole Squid314 archive, I had forgotten about this follow-up.
summa_tech · 16 days ago
A distant relative, no doubt, of Stanislaw Lem's "Automatthew's Friend" (1964). A perfectly rational, indestructible, selfless, well-meaning in-ear AI assistant. In the end, out of nothing but the deepest care for its owner's mental state in a hopeless situation, it advocates efficient and quick suicide.
Jun8 · 16 days ago
Compare/contrast the Whispering earring/LLM chat with The Room from Stalker, each one is terrifying in its aspect: One because it eventually coaxes you to become a shallow shell of yourself, the other by plucking an unexpected wish from the deepest part of your psyche. I wonder what the Earring would advise if one were to ask it if one should enter The Room.
djoldman · 16 days ago
> It is not a taskmaster, telling you what to do in order to achieve some foreign goal. It always tells you what will make you happiest....The earring is never wrong.

> There are no recorded cases of a wearer regretting following the earring’s advice, and there are no recorded cases of a wearer not regretting disobeying the earring. The earring is always right.

> ...The wearer lives an abnormally successful life, usually ending out as a rich and much-beloved pillar of the community with a large and happy family.

> Niderion-nomai’s commentary: It is well that we are so foolish, or what little freedom we have would be wasted on us. It is for this that Book of Cold Rain says one must never take the shortest path between two points.

The piece implies that

1. at least occasionally one should choose to do something one will regret.

2. not knowing what will make one happy is part of what makes one free.

I'm not sure I agree with these (it seems that 1. is a paradox) but it is an interesting thought experiment.

indoordin0saur · 16 days ago
I think it's less confusing when you consider the very first thing the earring says: "better for you if you take me off". The wearer should rationally always regret not following its advice, including that first thing.

I think the paradox is here, and it comes from cheeky use of misleading language:

> ...The wearer lives an abnormally successful life, usually ending out as a rich and much-beloved pillar of the community with a large and happy family.

The wearer doesn't really live any sort of life. Once it fully integrates with you your brain is mush, you're no longer experiencing anything. At some fuzzy point in there you've basically died and been replaced by the earring.

joshkel · 16 days ago
> at least occasionally one should choose to do something one will regret.

Not necessarily. My take was that the practice of choosing may well be more valuable than the harm of the occasional regretted choice.

munificent · 16 days ago
Statements that involve the future are always linguistically vague.

In your paradoxical sentence, "will regret" could be interpreted as either "know at that moment that they will regret" or "come to know after the fact that they regret it".

The former is a paradox, but the latter isn't.

As life advice, I think it works better when you consider it amortized over a collection of choices instead of a set of serial choices each of which it must be rigidly applied to: One should make a set of choices using a strategy that leads some of them to be likely to be regretful (but presumably without being able to predict ahead of time which ones will be).

nine_k · 16 days ago
> at least occasionally one should choose to do something one will regret.

Negative experience is crucial for learning, unfortunately. "If you never fail you don't try hard enough", etc. This is trivially understood in physical training: you have to get yourself exhausted to become stronger. It's much less of an accepted view in, so to say, mental training: doing thing that you later regret may teach you something valuable that always avoiding such decisions does not.

I do not necessarily support or reject this view, I'm just trying to clarify the point.

cjameskeller · 16 days ago
We are told:

>"It does not always give the best advice possible in a situation. It will not necessarily make its wearer King, or help her solve the miseries of the world. But its advice is always better than what the wearer would have come up with on her own."

I think one very simple explanation would be that this comes down to a matter of exploration vs exploitation. Since it is only giving "better" advice, and not even 'locally optimal', there is reason to favor exploring vs merely following the advice unquestioningly.

A more complex, but ultimately comprehensive answer, is that free will consists, at least in one aspect, in the ability not only to choose one's goals or means, but also what _aspect_ of those various options to consider "good" or "better".

And if one were to say that all such considerations ultimately resolve back to a fundamental desire to be "happy", to me, this seems to be hand-waving, rather than addressing the argument, because different people have different definitions of the "happy" end-state. If these differences were attributed fully to biology & environment, the story loses its impact, because there was never free will in the first place. If, while reading the story, we adopt a view that genuine free will exists, and hold some kind of agnosticism about the possible means by which that can be so, then it seems reasonable to attribute at least some of the differences in what the "happy" end-state looks like to the choices made by the people, themselves.

Given that kind of freedom, unless one has truly perfect knowledge (beyond the partial knowledge contained in the advice of the earring), the pursuit of one's goals seems to unavoidably entail some regrets. And with perfect knowledge, well... The kind of 'freedom' attributed, for example, to God by philosophers like Thomas Aquinas, is explicitly only analogous to our own, and is understood to be an unchanging condition, rather than a sequential act.

(As a final note: One might wonder what this 'freedom to choose aspects' approaches as an 'asymptotic state' -- that is, for an immortal person. And this leads to metaphysical concerns -- of course, with some things 'smuggled in' by the presumption of genuine freedom, already. Provided one agrees that human nature undeniably provides some structure to ultimate desires/"happiness", the idea of virtue ethics follows naturally, and from there many philosophers have arrived at similar notions of some kind of apotheosis as a stable end-state, as well as the contrary state of some kind of scattering or decay of the mind...)

Deleted Comment

Deleted Comment

adamgordonbell · 16 days ago
Wow, small world, I just made a podcast episode about the dangers of turning your brain off when using Agentic coding solution and referenced the whispering earring as my metaphor.

I feel like if you use the agentic tools to become more ambitious the you'll probably be fine. But if you just work at a feature factory where you crank out things as fast as you can AI coding is going to eat your brain.

Link: https://corecursive.com/red-queen-coding/#the-whispering-ear...

kybernetikos · 16 days ago
I'm not actually sure how horrifying this is. It sounds like it's just a better executive planner to achieve your goals. As long as they are still your goals, surely you'd want the best executive planner available. I would say it's the goals that are important, not the limited way in which I work out how to achieve them.

It would certainly be horrifying if I were slowly tricked into giving up my goals and values, but that doesn't seem to be what is happening in this story.

Perhaps if I were to put the earring on it would tell me it would be better for me to keep wearing it.

ToValueFunfetti · 16 days ago
You surrender your self in exchange for your goals. With the right goals, that could be a worthy sacrifice. But of course it is a sacrifice.

Imagine doing a crossword while a voice whispers the correct letter to enter for each cell. You'd definitely finish it a lot faster and without making mistakes. Crossword answers are public knowledge, and people still work them out instead of looking them up. They don't just want to solve them; they want to solve them theirselves. That's what is lost here.

kybernetikos · 16 days ago
This is associating the self with the thing that decides how best to achieve goals (the earring / the part of your brain that works out how to achieve a goal), while I'm saying that I think I would associate the self much more with the thing that decides what the goals are.

> they don't just want to solve them; they want to solve them theirselves. That's what is lost here.

I think in this story, the earring would not solve the crossword for you, if for some reason your goal was to solve the crossword yourself.

naasking · 15 days ago
> I'm not actually sure how horrifying this is. It sounds like it's just a better executive planner to achieve your goals. As long as they are still your goals, surely you'd want the best executive planner available.

The goals you form depend on your values, and your values are formed by learning from trial and error by what you like and what you regret. If the earring removes all chance of regret, then you also remove all chance of learning and all possibility of forming values or meaningful goals. You effectively erase yourself, hence why the brains were atrophied.

tacitusarc · 16 days ago
I think this ignores the internal conflict in most people’s psyche. The simplest form of this is long term vs short term thinking, but certainly our desires pull us in competing, sometimes opposite, directions.

Am I the me who loves cake or the me who wants to be in shape? Am I the me who wants to watch movies or who wants to write a book?

These are not simply different peaks of a given utility function, they are different utility functions entirely.

Soon after being put on, the whispering earring would go insane.

Deleted Comment

throw432189 · 16 days ago
Two points I liked:

1. I like that the first bit of advice is to take it off. It's very interesting that in this story very few people take its advice.

2. It recommends whatever would make you happiest in that moment, but not what would make the best version of yourself happiest, or what would maximize happiness in the long term.

Solving mazes requires some backtracking, I guess. Doing whatever will make you happiest in the moment won't make you happiest in the long run.