I got so hopefuly by your comment I showed it my current bug that I'm working on, I even prepared everything first with my github issue, the relevant code, the terminal with the failing tests, and I pasted it the full contents of the file and explained carefully what I wanted to achieve. As I was doing this, it repeated back to me everything I said, saying things like "if I understand correctly you're showing me a file called foo dot see see pee" and "I see you have a github issue open called extraneous spaces in frobnicator issue number sixty six" and "I see you have shared some extensive code" and after some more of this "validation"-speak it started repeating the full contents of the file like "backquote backquote backquote forward slash star .. import ess tee dee colon colon .." and so on.

Not quite up to excited junior-level programmer standards yet. But maybe good for other things who knows.

THIS is the thing I'm excited for with AI.

I'm someone that becomes about 5x more productive when I have a person watching or just checking in on me (even if they're just hovering there).

Having AI to basically be that "parent" to kick me into gear would be so helpful. 90% of the time my problems are because I need someone to help keep the gears turning for me, but there isn't always someone available. This has the potential to be a person that's always available.

At least you can theoretically stop sharing with this one. Microsoft was essentially trying to do this, but doing it for everything on your PC, with zero transparency.

Here's Google doing essentially the same thing, even more so that it's explicitly shipping your activity to the cloud, and this response is so different from the "we're sticking this on your machine and you can't turn it off" version Microsoft was attempting to land. This is what Microsoft should have done.

This is great! I viscerally dislike the "we're going to do art for you so you don't have to... even if you want to..." side of AI, but learning to use the tools to get the satisfaction of making it yourself is not easy! After 2 decades of working with 2D art and code separately, learning 3D stuff (if you include things like the complex and counterintuitive data flow of simulations in Houdini and the like) was as or more difficult than learning to code. Beyond that, taking classes is f'ing expensive, and more of that money goes to educational institutions than the teachers themselves. Obviously, getting beyond the basics for things that require experienced critique are just going to need human understanding, but for the base technical stuff, this is fantastic.

This comment is better than entire ad google just showed. Who is still pointing at the building with camera and asking what is this building?

Sounds interesting, but voice input isn't working for me there. I guess I'm too niche with my Mac and Firefox setup.

I tried this, shared a terminal, asked it to talk about it, and it guessed that it was Google Chrome with some webUI stuff. Immediately closed the window and bailed.

Get started documentation on Multimodal Live API : https://ai.google.dev/api/multimodal-live

I don't know what's not working but I get "Has a large language model. I don't have the capability to see your screen or any other visual input. My interactions are purely based on the text that you provide"

This'll be so fantastic once local models can do it, because nobody in the right mind would stream their voice and everything they do on their machine to Google right? Right?

Oh who am I kidding, people upload literally everything to drive lmao.

Published some more detailed notes on my explorations of Gemini 2.0 here https://simonwillison.net/2024/Dec/11/gemini-2/

Code execution is okay, but soon runs into the problem of missing packages that it can't install.

Practically, sandboxing hasn't been super important for me. Running claude with mcp based shell access has been working fine for me, as long as you instruct it to use venv, temporary directory, etc.

Question: Have you tried using this for video?

Alternately, if I wanted to pipe a bunch of screencaps into it and get one grand response, how would I do that?

e.g. "Does the user perform a thumbs up gesture in any of these stills?"

[edit: also, do you know the vision pricing? I couldn't find it easily]

> Some pelicans have white on their heads, suggesting that some of them are older birds.

Interesting theory!

> Remains to be seen how well they will be able to productize and market

The challenge is trust.

Google is one of the leaders in AI and are home to incredibly talented developers. But they also have an incredibly bad track record of supporting their products.

It's hard to justify committing developers and money to a product when there's a good chance you'll just have to pivot again once they get bored. Say what you will about Microsoft, but at least I can rely on their obsession with supporting outdated products.

  > Google is one of the leaders in AI and are home to incredibly talented developers. But they also have an incredibly bad track record of supporting their products.

With many research areas converging to comparable levels, the most critical piece is arguably vertical integration and forgoing the Nvidia tax.

They haven't wielded this advantage as powerfully as possible, but changes here could signal how committed they are to slaying the search cash cow.

Nadella deservedly earned acclaim for transitioning Microsoft from the Windows era to cloud and mobile.

It will be far more impressive if Google can defy the odds and conquer the innovator's dilemma with search.

Regardless, congratulations to Google on an amazing release and pushing the frontiers of innovation.

> and Google has been famously bad at getting people aligned and driving in one direction.

To be fair, it's not that they're bad at it -- it's that they generally have an explicit philosophy against it. It's a choice.

Google management doesn't want to "pick winners". It prefers to let multiple products (like messaging apps, famously) compete and let the market decide. According to this way of thinking, you come out ahead in the long run because you increase your chances of having the winning product.

Gemini is a great example of when they do choose to focus on a single strategy, however. Cloud was another great example.

BERT and Gemma 2B were both some of the highest-performing edge models of their time. Google does really well - in terms of pushing efficiency in the community they're second to none. They also don't need to rely on inordinate amounts of compute because Google's differentiating factor is the products they own and how they integrate it. OpenAI is API-minded, Google is laser-focused on the big-picture experience.

For example; those little AI-generated YouTube summaries that have been rolling out are wonderful. They don't require heavyweight LLMs to generate, and can create pretty effective summaries using nothing but a transcript. It's not only more useful than the other AI "features" I interact with regularly, it doesn't demand AGI or chain-of-thought.

> but hard to deny that their LLM models aren't really, really good though

Although I do still pay for ChatGPT, I find it dog slow. ChatGPT is simply way too slow to generate answers. It feels like --even though of course it's not doing the same thing-- I'm back to the 80s with my 8-bit computer printing thing line by line.

Gemini OTOH doesn't feel like that: answers are super fast.

To me low latency is going to be the killer feature. People won't keep paying for models that are dog slow to answer.

I'll probably be cancelling my ChatGPT subscription soon.

Well, compared to github copilot (paid), I think Gemini Free is actually better at writing non-archaic code.

>> hard to deny that their LLM models aren't really, really good though.

The context window of Gemini 1.5 pro is incredibly large and it retains the memory of things in the middle of the window well. It is quite a game changer for RAG applications.

Maybe I've just hit a streak of good outputs, but I've also been noticing the automatic gemini search when doing google searches to have been significantly more useful than it was previously.

About a year ago, I was saying that Google was potentially walking toward its own grave due to not having any pivots that rivaled OpenAI. Now, I'm starting to think they've found the first few steps toward an incredible stride.

Yet, google continues to show it'll deprecate it's APIs, Services, and Functionality at the detriment of your own business. I'm not sure enterprises will trust Google's LLM over the alternatives. Too many have been burned throughout the years, including GCP customers.

The fact GCP needs to have this page, and these lists are not 100% comprehensive is telling enough. https://cloud.google.com/compute/docs/deprecationshttps://cloud.google.com/chronicle/docs/deprecationshttps://developers.google.com/maps/deprecations

Steve Yegge rightfully called this out, and yet no change has been made. https://medium.com/@steve.yegge/dear-google-cloud-your-depre...

So far, for my tests, it has performed terribly compared to ChatGPT and Claude. I hope this version is better.

> seems to be getting the right results

> hard to deny that their LLM models aren't really, really good though

I'm so scarred by how much their first Gemini releases sucked that the thought of trying it again doesn't even cross my mind.

Are you telling us you're buying this press release wholesale, or you've tried the tech they're talking about and love it, or you have some additional knowledge not immediately evident here? Because it's not clear from your comment where you are getting that their LLM models are really good.

Oh wow, it supports directly specifying a Pydantic model as an output schema that it will adhere to for structured JSON output. That’s fantastic!

https://github.com/googleapis/python-genai?tab=readme-ov-fil...

its interesting that just as the LLM hype appears to be simmering down, DeepMind is making big strides. I'm more excited by this than any of OpenAI's announcements.

I looked carefully at the SDK earlier today - it does look very nice, but it is also a work in progress.

If you look at where talent is going, it's Anthropic that is the real competitor to Google, not OpenAI.

Are these benchmarks still meaningful?

Regarding TPU’s, sure for the stuff that’s running on the cloud.

However their on device TPUs lag behind the competition and Google still seem to struggle to move significant parts of Gemini to run on device as a result.

Of course, Gemini is provided as a subscription service as well so perhaps they’re not incentivized to move things locally.

I am curious if they’ll introduce something like Apple’s private cloud compute.

Yeah they've been slow to release end-user facing stuff but it's obvious that they're just grinding away internally.

They've ceded the fast mover advantage, but with a massive installed base of Android devices, a team of experts who basically created the entire field, a huge hardware presence (that THEY own), massive legal expertise, existing content deals, and a suite of vertically integrated services, I feel like the game is theirs to lose at this point.

The only caution is regulation / anti-trust action, but with a Trump administration that seems far less likely.

To play devil's advocate, the correct use of the word would be when multiple AIs are coordinating and handing off tasks to each other with limited context, such that the handoffs are dynamically decided at runtime by the AI, not by any routine code. I have yet to see a single example where this is required. Most problems can be solved with static workflows and simple rule based code. As such, I do believe that >95% of the usage of the word is marketing nonsense.

Versatile is far worse. It’s so broad to the point of meaninglessness. My garden rake is fairly versatile.

Agentic to me means that it acts somewhat under its own authority rather than a single call to an LLM. It has a small degree of agency.

I'm personally very glad that the word has adhered itself to a bunch of AI stuff, because people had started talking about "living more agentically" which I found much more aggravating. Now if anyone states that out loud you immediately picture them walking into doors and misunderstanding simple questions, so it will hopefully die out.

Need a general term for autonomous intelligent decision making.

Huh, all three words you mentioned as replacement are equally buzzwordy and I see them a lot in CVs while screen candidates for job interview.

Yeah I hate it when AI companies throw around words like AGI and agentic capabilities. It’s non sense to most people and ambiguous at best

agentic obviously means "not gentic" and gent is from the Latin for "people".

agentic == not people.

Quite sensible, really.

Take good care of your teeth, my friend. It's a new era in agentic factuality.

I use a Claude and Gemini a lot for coding and I realized there is no good or best model. Every model has it's upside and downside. I was trying to get authentication working according to the newer guidelines of Manifest V3 for browser extensions and every model is terrible. It is one use case where there is not much information or right documentation so every model makesup stuff. But this is my experience and I don't speak for everyone.

> For the more Java proficient, can someone explain why it may have provided this code:

To me that reads like it was trying to accomplish something like

    int[] current;
    while((current = queue.pop()) != null) {

I can't comment on why the model gave you that code, but I can tell you why it was not correct.

`queue.remove(0)` gives you an `int[]`, which is also what you were assigning to `current`. So logically it's a single element, not an iterable. If you had wanted to iterate over each item in the array, it would need to be:

``` for (int[] current : queue) { for (int c : current) { // ...do stuff... } } ```

Alternatively, if you wanted to iterate over each element in the queue and treat the int array as a single element, the revised solution is the correct one.

A tangent, but is there a clear best choice amongst those models for AOC type questions?

I asked Gemini 2.0 Flash (with my voice) whether it natively understands audio or is converting my voice to text. It replied:

"That's an insightful question. My understanding of your speech involves a pipeline first. Your voice is converted to text and then I process the text to understand what you're saying. So I don't understand your voice directly but rather through a text representation of it."

Unsure if this is a hallucination, but is disappointing if true.

Edit: Looking at the video you linked, they say "native audio output", so I assume this means the input isn't native? :(

These are not computer vision tasks…