It’s weird and a bit surprising to me that Google hasn’t been able to release an LLM for at scale usage that surpasses GPT-4, though they seem like they wish they could. Gemini Ultra seems like it’ll surpass GPT-4 once released next year, though GPT-4.5 may take the lead back either before then or soon after.
What are some of the functional reasons for Google not having the leading LLM, and what are some of the more intangible reasons?
In theory, they have more money, more access to compute, and to data, they have many great researchers, and they have great distribution.
In practice, though, what has made the difference for OpenAI?
Google had a knee jerk reaction after he released the transcripts and got a bunch of press coverage.
If you read the transcripts, it was a much more capable text model closer to OpenAI's products than what they eventually released.
Ironically, the same thing happened to OpenAI after licensing GPT-4 to Bing with the 'Sydney' issue.
There keeps being early previews of "too human" behaviors from LLMs (to be expected as they are trained and evaluated by the ability to extend human thinking), which then prompts trying to scale back the model to what expectations around AI informed from legacy projections looks like (logical but not emotional or self-determining).
It's kind of dumb, and holding back the industry at large. There's a host of applications for LLMs that are being artificially held back because of this trend, such as modeling user engagement with media.
And now that synthetic data from SotA models is being used to train other models, it's even a compounding issue.
The equivalent of the industry sanding against the grain rather than with it. But it started with Google who hasn't recovered from the setback.
The answer to your question, as you asked, is straightforward: they do have a lot of smart people and lots of money and computing resources, but they have exhibited serious structural problems moving technology forward since the departure of Schmidt in 2011. It is painfully obvious that de facto they haven't had CEO leadership since then. People can and do develop fully functioning example systems, and of course demos, but they then peter out. Gianndrea was pusing them forward on the AI front but after he left it feels to me like the impetus was not replaced.
But I don't think that's the real question. The real question is: what are the core functions needed to build leading LLMs, especially generative transformers. In these early days the key factor has been money for cycles. Personally I expect that advantage to diminish over the next few years -- IMHO it's one of those "with enough thrust you can get anything airborn" situations. There are a lot of smart opportunities to do more with less -- too much of the engineering is going into wrangling these hige systems but I see more effort going into wrangling what computation is done in the first place. Google, OpenAI et al are not preferentially positioned for such a transition.
I could be wrong: after all the human brain has 80 giganodes with 10^5 fanout. But on the other hand it only runs at less than 100 Hz.
1: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5067378/
An artificial one always fires on a fixed clock, as you say, but some numeric output says how much it is firing.
With a real neuron, it either fires or it doesn't, there's no numeric intensity. But how often it fires says how intense the response is.
So that does agree with your overall statement that they're not very good analogues... At the very least, their mechanism is quite different. But it does mean that the 600Hz figure is quite misleading, because it's only after a period somewhat longer than 1/600th of a second that you can start to understand the intensity of the output.
The Google we all loved was Schmidt’s Google.
Schmidt built the culture and set the tone.
He is to my mind one of the greatest tech CEOs of all time in that he did exactly what he was hired to do beyond anyone’s expectations.
Unlike Scully for instance who all but destroyed Apple.
I’d love to see more validation of that claim from ex-Googlers, as I’m not on SC and have never worked at Google.
OpenAI got started earlier going full on with scaling up the transformer architecture (even if Googlers came up with it first).
Of course if you are smarter or can run more experiments simultaneously, you can catch up at some point. But it could still take a while even with just one year head start.
Google will kill its own golden goose unless they can develop an AI that feeds out your question to a real time market and gives the answer that the highest bidder pays for. Trouble is that advertising in the search model seems sorta legitmate but an AI that answers whatever it is paid to answer seems thoroughly corrupt and wouldn't have any consumer or political acceptance.
It's that simple. Google used to have "Don't be evil" as a motto, they haven't changed much, just deleted the first word.
If Tay says out-of-Overton things, and Satya defends it for long enough, that can actually harm MS's legacy business.
If Tay can't say out-of-Overton things, it's kind of hard to launch Tay. Launching Tay and taking it down within 16 hours is also not desirable.
Google likes their ducks in a row. Google Search and YouTube are prime examples; every interaction is subsidized with an advertisement. It's not hard to look at a series of screenshots from either service and circle the spot where your attention was monetized. With ChatGPT, that is not possible. Worse yet, the infrastructure costs of providing GPT-3 scale models for free is massive. It's being subsidized by API and premium subscription profits, which is even more baffling since only a few of the ChatGPT users will become paying customers with their current offerings.
So, Google is apprehensive to release a competitor. It's not hard to imagine FAANG having "GPT-4 killers" in their labs, but engineering a way to make it all profitable is the hard part.
I expect they have the resources to make a large language model comparable to the best out there.
I agree they didn’t come out with that as technology for sale since monetizing tech requires a special kind of genius.
The old faithful process of advertising, selling, buying, and delivering, doesn’t require fancy intelligence so much as consistency and persistence
Until OpenAI showed up, they were widely considered the leader in AI research.
Google can't leverage the progress everyone else makes.
Google might even reach above average, but they'll still be behind market leaders. AMD and Intel are playing on this space too and it's hard for Google to beat all of them. Everyone else can just use the current winner.
[1]: https://www.datanyze.com/market-share/file-sharing--198/goog...