Readit NewsI can't wait to play with this but I can't even imagine how expensive it must be. They're training in full resolution and can generate up to a minute of video.
Seeing how bad video generation was, I expected it would take a few more years to get to this but it seems like this is another case of "Add data & compute"(TM) where transformers prove once again they'll learn everything and be great at it
Or is it the deal with Microsoft for cloud services making it cheap?
Or are they just operating at a massive loss to kill off other competition?
Or something else?
If that was the case, it would make it difficult to generalize these concepts.
I don’t feel like the points made here align with any insight about the workings of LLMs. The fact that, as a human, I “wouldn’t know where to start” when asked to add two numbers without doing any addition doesn’t apply to computers (running predictive models). They would start with statistics over lots of similar examples in the training data. It’s still remarkable LLMs do so well on these problems, while at the same time doing somewhat poorly because they can’t do arithmetic!
I suspect most of this is due to tokenization making it difficult to generalize these concepts.
There are some weird edge cases though, for example GPT-4 will almost always be able to add two 40 digits number but it is also almost always wrong when adding a 40 digit and 35 digit number.
Maciej's larger point is that the AI faces tons of very difficult problems in escaping its physical constraints. It's simplistic to wave hands and say "the AI is super duper smart and will have no difficulty hacking all computer systems, inventing and manufacturing swarms of unbeatable nanobots, etc. without being detected or resisted".
It seems to me that the core conceit of AI doomerism is that sufficient intelligence can overcome all barriers with some plan that is so smart, people would never think of it. This is much less plausible than believers take it to be. In mathematics alone, it is very easy to come up with a problem that the collective mathematical ingenuity of the entire human race is helpless before for decades, centuries, or longer.
However, seeing how excited Palantir is with their war assistant LLM , the US testing autonomous fighter jets a few months ago, etc. I think there's a decent chance that AI won't even have to break out of its constraints. It's pretty much guaranteed people are going to do the obviously dumb thing and give it capabilities it shouldn't have or is not equipped to deal with safely.
Dead Comment
LLMs are not sentient. They have no agency. They do nothing a human doesn't tell them to do.
We may create actual sentient independent AI someday. Maybe we're getting closer. But not only is this not it, but I fail to see how trying to license it will prevent that from happening.
AutoGPTs and the likes are much overhyped (it's early tech experiments after all) and have not produced anything of value yet but having dabbled with autonomous agents, I definitely see a not so distant future when you can outsource valuable tasks to such systems.
We've been doing this forever with everything. Building tools is what makes us unique. Why is building what amounts to a calculator/spreadsheet/CAD program for language somehow a Rubicon that cannot be crossed? Did people freak out this much about computers replacing humans when they were shown to be good at math?
We've already crossed it and I believe we should go full steam ahead, tech is cool and we should be doing cool things.
> Did people freak out this much about computers replacing humans when they were shown to be good at math?
Too young but I'm sure they did freak out a little! Computers have changed the world and people have internalized computers as being much better/faster at math but exhibiting creativity, language proficiency and thinking is not something people thought computers were supposed to do.
I view it as being OSS or postgresql dev that happens to work at microsoft. I've been doing the former for much longer (starting somewhere between 2005 and 2008, depending on how you count) than the latter (2019-12).