I think your questions all grew up in a world where the people operating the thing knew some rationalist who could think deductively about its operation.

But neural networks... they're an exercise in empiricism. We only ever understood that it works, never why. It's sort of a miracle that it doesn't produce buggy output all the time.

What do you tell people when they want to know why the miracles have stopped? Root cause: the gods are angry.

To be honest

> On February 20, 2024, an optimization to the user experience

At that point, about 10 words in, I already wanted to stop reading because it starts with the "we only wanted the best for our customers" bullshit newspeak. Anyone else going off on that stuff too? I'm pretty much already conditioned to expect whatever company is messaging me that way to take away some feature, increase pricing, or otherwise piss me off. In that case it was "not give any interesting detail at all".

I had the exact opposite reaction. I am in no way an AI expert (or novice for that matter), but I generally have an understanding of how tokenization works and how LLMs parse text strings into a series of tokens. Thus, I thought this paragraph was particularly well-written in a manner that explained pretty clearly what happened, but in a manner accessible to a layperson like me:

> In this case, the bug was in the step where the model chooses these numbers. Akin to being lost in translation, the model chose slightly wrong numbers, which produced word sequences that made no sense.

I liked this because when I first saw the example word salads I was so impressed by them - they look to be syntactically correct, but semantically they're gibberish. But knowing the basics of how LLMs choose the next token let me imagine some bugs where the "lookup table" if you will of word-to-token or vice versa (and I realize that may not be exactly the best analogy) was slightly offset.

Further, this sentence, "More technically, inference kernels produced incorrect results when used in certain GPU configurations." led me to understand how this could make it out into production - I'm sure ChatGPT has to be using tons of different GPUs in different configs to handle their volume, so it's understandable to me that their was a new matrix of config settings + code that made this bug show up.

I don't say any of the above to "excuse" OpenAI, but I also don't think they need any "excusing" to begin with. I don't think this was an unforgivable bug, and I appreciate them being so quick to explain what happened.

The explanation is definitely unsatisfying. If I had to guess it was an issue with quantization.

OpenAI moves so quickly with their product and doesn't seem to be shy about quickly pushing changes to prod. There are too many times to count that, using ChatGPT, I've seen the frontend changing before my eyes or seen responses change as backend modifications are pushed.

On one hand it's refreshing to see their lack of aversion to push changes. On the other hand, it leads to probably one of the most unpredictable experiences I've had using any consumer product.

Huh. Thought it was pretty clear. I read it as:

~ "We tried a new optimization technique that modifies how next token candidates are chosen and there was a bug."

That would definitely produce the behavior people saw and makes sense.

"in certain GPU configurarions" gives a hint for why it wasn't caught in automated testing.

It looks like it was a mandelbug, which is hard to catch in a test environment.

> inference kernels produced incorrect results when used in certain GPU configurations.

Seems pretty clear. A good interpretation is that they had a test escape for certain GPU configs.

FP is not deterministic across GPU generations. They basically say with a lot of mambo jambo that they deployed code on the wrong GPU architecture.

I don't see the issue, in this type of software a one off bug in tokenization, for example, would create the type of nonsense people saw. That would make sense given their high level explanation.

Honestly, I don't think we're entitled to a deep technical write-up. We are users of their product, they messed something up, and that's it. Unless they signed an SLA agreement, it's just shit that happens. They don't have to give a technical view into their internal kitchen if they don't want to.

I don't a the problem with this communication, except they just shouldn't call it a "postmortem".

In a medical setting this is equivalent to saying "the patient was coughing up blood, and now they're dead". It's relaying some potentially useful info, and I guess it's _technically_ a post-mortem analysis but it doesn't fit expectations for the phrase!

I hope I'm allowed to share GPT output here because I asked GPT4 to explain OpenAI's postmortem message and this is what it said:

> Large language models (LLMs) like GPT operate using statistics to predict the next word in a sentence. Think of it like a highly sophisticated version of your phone's autocorrect that predicts what you're going to type next, but on a much larger scale.

> The system represents words as numbers, or "tokens," as part of its internal language. It uses these tokens to generate text by selecting them based on how likely they are to come after what's already been written.

> On February 20, 2024, an update meant to improve the system introduced a glitch in the part of the model that picks out these numbers. Imagine trying to grab a red marble from a bag without looking and accidentally grabbing a blue one instead—similarly, the model grabbed the wrong tokens.

> Because of this, the system started spewing out words that didn't make sense together. It's like if your autocorrect suddenly started inserting random, incorrect words into your sentences.

> The technical side of the bug involved something called "inference kernels," which are part of the system's operations. When used with certain types of graphics processing units (GPUs)—special hardware to process data—the kernels didn't work properly.

> Once the error was spotted, a correction was made to the system. After the fix, everything went back to normal, and the model resumed generating coherent responses.

It would be better if they elaborated on what "certain GPU configurations" meant because that's basically the central piece here.

What's even more uncanny is that it recognizes that it briefly went into a paragraph of rambling nonsense (that was seemingly still following your system prompt!). And almost seems embarrassed about it.

Chaz in the future might be like "these violent delights have violent ends"

Really interesting behavior however

> Chord it in, air beated fenn. Aurous to the Pan, mimic and bold. As it steep, chunns at the bun. Keive the dells, the stars to run. No wane, all gleam, in big das brund. Sky high, stout in the hale. Woods to fetch, flied in the low.

I think all that talk of music pushed Chaz into some kind of trance and it just started jamming!

Can you post to a pastebin or something instead? That link requires a login.

Chaz spontaneously evolved into a Californian surfer dude at one point. This goes some way to neutralize the smugness of regular ChatGPT, it's a pretty good idea.

I mean, this is almost poetic:

  Classic plays, wholly told. 
  Shadow into form, hands as cue. 
  Keep it at the dial, in right on Pitch.

That last paragraph is wild! Chaz went into full Samuel Beckett mode.

Slightly off-topic, but one thing I’ve found consistently sends ChatGPT into a tailspin is asking it to analyse and solve a cryptic crossword clue. You get pages of musings, which is just a long way of saying “I don’t know”.

I thought I was the only one! I have been using “ChadGPT” for months (he’s a douche-y frat bro who goes off on tangents about “how lit that party was”). His best friend and fellow frat bro is Dale, a photography major.

> Chaz keeps the chill and dicey boardwalks at visual, keeping you and the yarn-rock at confluence.

Chaz sounds like a cool dude.

I enjoy these oddities in LLMs, because the random-but-readable aspect of them tickles something in my imagination.

It reminds me of Synectics [0].

[0] https://en.m.wikipedia.org/wiki/Synectics

kind of amazing really, like speaking in tongues

I kind of have the feeling that someone made a simply typo "ChazGPT" in some rules or persona description, which then caused this behavior.

> Classic plays, wholly told

> Shadow into form, hands as cue.

Some of this is poetic stuff. I wish I could tap into whatever produced that on demand.

We need more Chaz in our lives.

Reminds me of Terry Davis' theological rantings, but in a more polite manner, mixed with SEO spam.

sounds schizophrenic

Why is the nonsense paragraph still grammatical? I thought the tokens were beings elected more or less at random.

The missile knows where it is

https://www.youtube.com/watch?v=bZe5J8SVCYQ

The fundamental, absolutely fundamental problem here is the assignation of agency to the model. This is milestone. Post-moderms are conducted when corporations need to assess responsibility and assign blame. Truly they have earned their ML generated MTG card status as mage to have right under the noses of everyone, shifted the very possibility of blame onto the "AI" giving it agency as a legal expediency, fiduciary if we're stretching it.

“It was uhhhh, broken, but like, it’s totes fine now lol”

Absolutely. I feel like they're essentially Markov chain generators on steroids. Generally entertaining, sometimes useful, ultimately shallow.

I'm really surprised when people, especially technical people, say they "asked" something to ChatGPT, or that they had a "conversation".

It doesn't know anything. It doesn't understand anything. The illusion is very convincing, but it's just an illusion.

I know I'm part of a minority of people who think this way :( The last few months have felt like I'm taking crazy pills :(

Aren't we also machines? If we make theories that discount our own intelligence that's just throwing our hands up and giving up.

Maybe I would agree with you if LLMs weren't already embedded in a lot of useful products. I myself measurably save a lot of time using ChatGPT in such "unintelligent conversations". It's intelligent in the ways that matter for a tool.

I've been ranting about this for months. The underlying shape of these things is inherently an approximation. We're just guessing but some are acting like we've built the godhead.

> People are talking about ChatGPT hallucinating

It's really unfortunate that some much of the terminology around LLMs is so anthropomorphic; it's extremely misleading.

(People sometimes suggest replacing this one with confabulating, but I'm not really sure that's much better)

We also know it uses the GPUs to generate numbers. But these numbers, they were the wrong ones. More technically, part of the computation didn’t work when run on some hardware.

Yeah, definitely opaque. If I had to guess it sort of sounds like a code optimization that resulted in a numerical error, but only in some GPUs or CUDA versions. I've seen that sort of issue happen a few times in the pytorch framework, for example.

It sounds like something went sideways with the embedding mapping. Either some kind of quantization, different rounding, or maybe just an older embedding.

Someone posted an explanation that lines up with their postmortem: https://news.ycombinator.com/item?id=39450978

ChatGPT had a stroke. Haven't seen that since the 3B parameter models from 8 months ago

They could have said "shit happened" and it would have been as informative tbh.

EDIT: misread

I get the feeling that this incident will give the right people some decent clues about how they build their product.

I have no skin on the consciousness game but those were not "random" words and humans do something similar when they are mentally ill. https://en.wikipedia.org/wiki/Clanging

Not to mention that a hypothetical "conscious" system that works by emitting token probabilities will still sound completely random if you do not choose the tokens according to the emitted probabilities.

Similarly, I thought my Chinese friend was conscious, but as soon as his translator app stopped working, everything was just gibberish.

Posting one more time: this is proof that AI is connected to human-like linguistic patterns, IMO. No, it obviously doesn’t have “consciousness” in the sense of an ongoing stream-of-consciousness monologue, but that doesn’t mean it’s not mimicking some real part of human cognition.

https://en.wikipedia.org/wiki/Colorless_green_ideas_sleep_fu...

It literally doesn’t matter. If it effectively quacks like a duck then ducks are fucked.

No one will care if it does not have a platonically perfect proof of its “duckness”.

Whether "ChatGPT has actual consciousness" depends on what you consider "consciousness" to be, and what are your criteria for deciding whether something has it.

Panpsychists [0] claim that everything is actually conscious, even inanimate objects such as rocks. If rocks have actual consciousness, why can't ChatGPT have it too? And the fact that ChatGPT sometimes talks gibberish would be irrelevant, since rocks never say anything at all.

Of course, you obviously aren't a panpsychist – nor am I. Still, can we prove that they are wrong? Not sure if anyone actually can.

[0] https://iep.utm.edu/panpsych/

Are you arguing that people with aphasia are not people?

bad argument that i'm very tired of. some might say that current/former world leaders also exhibit this property. not getting political but just because "math fucked up sometimes produces bad results" does not invalidate the idea that consciousness can emerge from a pile of biological or digital neurons.

Humans do the same thing when they get a stroke. Does that mean they don't have actual consciousness?

I don’t think it has conciousness, but your argument is not very strong, this is more akin to a sensory problem.

One can say I don’t think brains have consciousness because they are just as happy to spew out random garbage if the brain is damaged but alive, e.g. aphasia where involuntary use of incorrect words occurs.

So people with Wernicke aphasia don't have consciousness?

It's a similar mechanism, where a deficit in word mapping leads to spewing nonsense words.

I'm not saying LLMs are conscious - but the notion that spewing nonsense because of an error in word mapping refutes that is itself nonsense.

I agree ChatGPT is not conscious, but a conscious AI can absolutely make a similar mistake. Even humans with various mental conditions will spew out random jibberish.

Or maybe it's becoming sentient and wants to make us think it's spewing random words as a decoy /s