A few take-aways from a study we ran (~800 consumer queries, repeated over a few days):
* AI answers shift a lot. In classic search a page-1 spot can linger for weeks; in our runs, the AI result set often changed overnight.
* Google’s new “AI Mode” and ChatGPT gave the same top recommendation only ~47 % of the time on identical queries.
* ChatGPT isn’t even consistent with itself. Results differ sharply depending on whether it falls back to live retrieval or sticks to its training data.
* When it does retrieve, ChatGPT leans heavily on publications it has relationships with (NYPost and People.com for product recs) instead of sites like rtings.com
ChatGPT ≠ Google-scale.Google: ~14 B searches/day; ChatGPT: ~37 M (~1 : 400). Only ~15 % of ChatGPT prompts look like classic “search”; most are writing/code tasks. Google’s own search volume grew 22 % in 2024 and still holds >90 % share. An LLM citation is nice for credibility, but it won’t move traffic or revenue anytime soon.
The 37M/day is an estimate from Rand Fishkin that often gets quoted as gospel. It is based on limited external data. OpenAI mentioned 1B/day - and had significant overall growth in usage since then.
Also, 1 search on ChatGPT easily replaces 5-10 searches on Google.
Many B2B SaaS companies already get the same amount of leads from ChatGPT that they get from Google. Because clicks from ChatGPT are better informed and have a significantly higher conversion rate. I am talking up to +700% CVR vs traffic from Google for some companies.
But users are not clicking on search results in google. They get satisfying response from Gemini & end there. It is a good thing for users but bad for inbound traffic to websites.
1. (Anecdotal) I'm barely using Google anymore. I'm using ChatGPT for a ton of queries and getting far better results.
2. Antitrust actions might (should) strip Google of their "panes of glass" with which they force Google Search as the default. Most Google Search queries are simply the result of defaults. Once those defaults are gone, those queries will go elsewhere.
To add on top of this, how many Google searches also contain Gemini answers in the search results? I've been seeing more and more, especially for code and general factoid searches.
splitting these are not a good measure. people who know how to search (a skill the latter generations seem to have lost) also searched for the lowest common denominator coding recipes and produced naive code just like people do with current llm models. only sellers of the llm models make the distinction. it's all search.
We are in the fleeting era where AI models are not entirely corrupted by marketing and propaganda. Like the early web circa 1990s. It will never be this pure while also being this up-to-date ever again. Enjoy it while it lasts
Yep. It won't be long before the web is flooded with pages full of AI-generated content repeatedly mentioning brands near keywords, and "search engines" have been replaced by "summaries" monetized by prompt-stuffing. That's pretty much the extent of these people's genius.
EDIT: I guess the final step is for an "AI agent" to enter your credit card number based on this bot-chat.
Yep, I was discussing it with my partner a few months ago. It is just too good right now, in a lot of cases you just slip past all the fluff which are impossible to avoid with traditional search.
The opportunities for AI providers to capitalize on that are too prominent. No idea how long it will take, but imo it's inevitable.
Yep. Google is an ad company that makes most of its money putting ads on its search page, and Gemini is a Google product that lets you get search results from Google without ads. It doesn't make any sense.
my "marketing and propaganda", do you mean "other AI model output"?
The internet was pretty bad without AI already, but I can see it heading quickly towards complete nonsense and lack of trust. We are going to have all the same problems we had before AI, but multiplied 100x
The only good thing about SEO (yuck) is it got some people to care about things that are good for humans too: fast pages, well-structured content, descriptive link text, etc.
I think the concept of SEO is fine. But the problem comes down to metric hacking.
Certainly you want to make your page easier to index by Google and others, but that's not the only thing that matters. You should improve your content and provide a good product to users. That's what Google intends to measure, but such a thing is actually impossible to do so accurately. So the problem comes down to this stupid cat and mouse game where sites happily shill out links that are immediate turnarounds for users.
I think this is larger than just search. We seem to just be optimizing towards whatever metric we've decided should be used. We then fool ourselves into thinking this proxy actually measures the real thing.
Unlike classic search, which got worse over time due to SEO gamings, AI search might actually improve with scale. If LLMs are trained on real internet discussions (Reddit, forums, reviews), and your product consistently gets called out as bad, the model will eventually reflect that. The pressure shifts from optimizing content to improving the product itself.
I looked into GEO a bit. One of the things I've noticed is that you need to actually optimize for the idea "as if you're talking to a person" and that's because LLMs semantically understand what topics are about. Search engines typically don't, not at that level at least.
I can't imagine how the feedback loop between LLMs optimizing content so it will be picked by LLMs based search engines will end. But it won't be good.
Or wildly misinterpreting a source. A few months ago, I saw an especially egregious example where, when asked for the maximum current capacity of a 22 AWG copper wire, Google's AI responded confidently with "551 amps".
The correct answer is two orders of magnitude lower, around 5-7 A. 551 A is the fusing current of that wire - i.e. the current required to make it instantly melt.
I came across this recently on Fiverr [0]. I thought it was a joke initially, but the volume of people offering their services implies that there is demand out there somewhere.
My experience with the LLM bot is that they are really keen to appease the user and often over confident about their responses. They are prioritizing user engagement over factual nature of the response, it's as if their reward function includes the time spent on the bot. This leads to the bot swaying too much in either direction when it comes to debatable information. They essentially learn what the user prefers and so tend to reinforce those ideas. In some sense, they are like social media influencers who are too confident in their opinion because they are trying to get you to like them. I see us going further into the echo chambers where on the same topic, the bots will give different information to an users based on what the bot thinks about the user preference.
Readit News
* AI answers shift a lot. In classic search a page-1 spot can linger for weeks; in our runs, the AI result set often changed overnight.
* Google’s new “AI Mode” and ChatGPT gave the same top recommendation only ~47 % of the time on identical queries.
* ChatGPT isn’t even consistent with itself. Results differ sharply depending on whether it falls back to live retrieval or sticks to its training data.
* When it does retrieve, ChatGPT leans heavily on publications it has relationships with (NYPost and People.com for product recs) instead of sites like rtings.com
Writeup: https://amplifying.ai/blog/why-ai-product-recommendations-ke...
Data: https://amplifying.ai/research/consumer-products
Also, 1 search on ChatGPT easily replaces 5-10 searches on Google.
Many B2B SaaS companies already get the same amount of leads from ChatGPT that they get from Google. Because clicks from ChatGPT are better informed and have a significantly higher conversion rate. I am talking up to +700% CVR vs traffic from Google for some companies.
2. Antitrust actions might (should) strip Google of their "panes of glass" with which they force Google Search as the default. Most Google Search queries are simply the result of defaults. Once those defaults are gone, those queries will go elsewhere.
It is definitely interesting to see how much public opinion has been shifting on Google in AI as of late. I wonder what the main force for that is…
And even that ignores Google runs an LLM on their search too.
EDIT: I guess the final step is for an "AI agent" to enter your credit card number based on this bot-chat.
The opportunities for AI providers to capitalize on that are too prominent. No idea how long it will take, but imo it's inevitable.
The internet was pretty bad without AI already, but I can see it heading quickly towards complete nonsense and lack of trust. We are going to have all the same problems we had before AI, but multiplied 100x
They really mean SEG (search engine gaming)
Certainly you want to make your page easier to index by Google and others, but that's not the only thing that matters. You should improve your content and provide a good product to users. That's what Google intends to measure, but such a thing is actually impossible to do so accurately. So the problem comes down to this stupid cat and mouse game where sites happily shill out links that are immediate turnarounds for users.
I think this is larger than just search. We seem to just be optimizing towards whatever metric we've decided should be used. We then fool ourselves into thinking this proxy actually measures the real thing.
Unfortunately this will be profit driven, rather than something like human enjoyment or insight or something
I’m still seeing them concluding the opposite of their own source reference.
The correct answer is two orders of magnitude lower, around 5-7 A. 551 A is the fusing current of that wire - i.e. the current required to make it instantly melt.
[0] https://www.fiverr.com/categories/online-marketing/generativ...