Readit NewsThat's why LLM outputs that get cut off mid-stream require the end user click the "retry" button and not the, "re-send me that last output" button (which doesn't exist).
I would imagine that a simpler approach would be to simply make the last prompt idempotent... Which would require caching on their servers; something that supposedly isn't happening right now. That way, if the user re-sends the last prompt the server just responds with the same exact output it just generated. Except LLMs often make mistakes and hallucinate things... So re-sending the last prompt and hoping for a better output isn't an uncommon thing.
Soooo... Back to my suggested workaround in my other comment: Pub/sub over WebSockets :D
It's yet another system that needs some DRAM though. The good news is that you can auto-expire the queued up responses pretty fast :shrug:
No idea if it's worth it, though. Someone with access to the statistics surrounding dropped connections/repeated prompts at a big LLM service provider would need to do some math.
It's the inevitable peak of the venture capital pipeline, just this time it isn't individual industries (e.g. taxis with Uber, hotels with AirBnB) getting squeezed out by unsustainable pricing - it's the economy at large that's suffering this time.
And it's high time for us as a society to put an end to this madness. End the AI VC economy before it ends our economy.
I think regulating too hard here would result in black markets and gamblers becoming more vulnerable to bad actors.
that story is flawed for a lot of reasons, but it's interesting to explore what happens if death is essentially conquered.
it's hard to judge whether or not society as depicted in that story stagnated.. but it was wholly different.
If you want to complain about tech companies ruining the environment, look towards policies that force people to come into the office. Pointless commutes are far, far worse for the environment than all data centers combined.
Complaining about the environmental impact of AI is like plastic manufacturers putting recycling labels on plastic that is inherently not recycleable and making it seem like plastic pollution is every day people's fault for not recycling enough.
AI's impact on the environment is so tiny it's comparable to a rounding error when held up against the output of say, global shipping or air travel.
Why don't people get this upset at airport expansions? They're vastly worse.