Been thinking about this a lot [1]. Will this fundamentally change how people find and access information? How do you create an experience so compelling that it replaces the current paradigm?
The future promised in Star Trek and even Apple's Knowledge Navigator [2] from 1987 still feels distant. In those visions, users simply asked questions and received reliable answers - nobody had to fact-check the answers ever.
Combining two broken systems - compromised search engines and unreliable LLMs - seems unlikely to yield that vision. Legacy, ad-based search, has devolved into a wasteland of misaligned incentives, conflict of interest and prolifirated the web full of content farms optimized for ads and algos instead of humans.
Path forward requires solving the core challenge: actually surfacing the content people want to see, not what intermiediaries want them to see - which means a different business model in seach, where there are no intermediaries. I do not see a way around this. Advancing models without advancing search is like having a michelin star chef work with spoiled ingredients.
I am cautiously optimistic we will eventually get there, but boy, we will need a fundamentally different setup in terms of incentives involved in information consumption, both in tech and society.
I agree that this is the core question, but I'd put it as: Who Gets To Decide What Is True?
With a search paradigm this wasn't an issue as much, because the answers were presented as "here's a bunch of websites that appear to deal with the question you asked". It was then up to the reader to decide which of those sites they wanted to visit, and therefore which viewpoints they got to see.
With an LLM answering the question, this is critical.
To paraphrase a recent conversation I had with a friend: "in the USA, can illegal immigrants vote?" has a single truthful answer ("no" obviously). But there are many places around the web saying other things (which is why my friend was confused). An LLM trawling the web could very conceivably come up with a non-truthful answer.
This is possibly a bad example, because the truth is very clearly written down by the government, based on exact laws. It just happened to be a recent example that I encountered of how the internet leads people astray.
A better example might be "is dietary saturated fat a major factor for heart disease in Western countries?". The current government publications (which answer "yes") for this are probably wrong based on recent research. The government cannot be relied upon as a source of truth for this.
And, generally, allowing the government to decide what is true is probably a path we (as a civilisation) do not want to take. We're seeing how that pans out in Australia and it's not good.
> To paraphrase a recent conversation I had with a friend: "in the USA, can illegal immigrants vote?" has a single truthful answer ("no" obviously)
Er, no, the meaning of the question is ambiguous, so I'm not sure "has a single truthful answer" is accurate. What does "can" mean? If you mean "permitted", then no. But if you mean can they vote anyway and get away with it? It's clearly happened before (as rare as it might have been), so technically the answer to that would be be yes.
Neal Stephenson's book _Fall, or; Dodge in Hell_, from 2019, dedicates many words to this concept. Briefly summarized, it explores a post-truth-world, describing the world where people could agree on the truth as a narrow time-slice in history. People have their own individual internet filters, and the USA becomes divided into Afghanistan-etc-like tribes, each an echo chamber. (Ameristan)
> A better example might be "is dietary saturated fat a major factor for heart disease in Western countries?". The current government publications (which answer "yes") for this are probably wrong based on recent research. The government cannot be relied upon as a source of truth for this.
I know it was just an example, but actually no, the role of dietary saturated fat as a factor for heart disease remains very much valid. I’m not sure which recent studies you’re referring to, but you can't undo over 50 years of research on the subject so easily. What study were you thinking about?
> To paraphrase a recent conversation I had with a friend: "in the USA, can illegal immigrants vote?" has a single truthful answer ("no" obviously). But there are many places around the web saying other things (which is why my friend was confused). An LLM trawling the web could very conceivably come up with a non-truthful answer.
There are many jurisdictions where some illegal immigrants (Dreamers) are allowed to vote, including New York City[1].
I was thinking something similar. I like the google auto-ai summary for little trivia facts and objective things (who played so and so in the movie? How heavy is a gallon of milk?) this is stuff that is verifiable and could theoretically just be queried from some sort of knowledge base.
But for anything remotely subjective, context dependent, or time sensitive I need to know the source. And this isn’t just for hot button political stuff — there’s all sorts of questions like “is the weather good this weekend?”, “Are puffy jackets cool again?”, “How much should I spend on a vacation?”
What is truth anyway? I see it as a quicker version of browsing to web to get a summary of what people say. As you said, with search you get a bunch of websites where strangers talk about a certain topic. You read a dozen and see if they agree with each other or if they sound legit.. There is just a huge overlap between what we consider true and what (an overwhelming majority of) people agree on. A lot of things are reduced to the consensus. If you ask a non-obvious question, you usually get an answer "a lot of people you consider trustworthy dedicated some time studying this question and agreed that the answer is X". But then people can be wrong, a big group of people can be wrong, people can be bribed or perhaps you don't actually trust these people that much. The internet can't tell you the truth. LLM can't tell you the truth. But they can summarize what other people in the world say on this subject.
> I agree that this is the core question, but I'd put it as: Who Gets To Decide What Is True?
Well, outside of matters of stark factuality (what time does the library close?, what did MSFT close at?), many things people may be "searching for" (i.e. trying to find information about) are more in the realm of informed opinion and summary where there is no right or wrong, just a bunch of viewpoints, some probably better informed than others.
I think this is the value of traditional search where the result is a link to a specific page/source whose trustworthiness or degree of authority you can then judge (e.g. based on known reputation).
For AI generated/summarized "search results", such as those from "ChatGPT Search" (awkward name - bit like a spork), the trustworthiness or degree of authority is only as good as the AI (not person) that generated it, and given today's state of technology where the "AI" is just an LLM (prone to hallucination, limited reasoning, etc), this is obviously a bit of an issue...
Even in the future, when presumably human level AGI will have made LLMs obsolete, I think it'll still be useful to differentiate search from RAG AGI search/chat, since it'll still be useful to know the source. At that point the specific AGI might be regarded as a specific person, with it's own areas of expertise and biases.
The name "ChatGPT Search" is very awkward - presumably they are trying to position this as a potential Google competitor and revenue generator (when the inevitable advertisements come), but at the end of the day it's just RAG.
> It was then up to the reader to decide which of those sites they wanted to visit,
With current search engines, Google decides for you ("helped" by SEO experts failing over themselves to rank higher because their revenue directly depends on it). In theory, you could go an read a few dozen pages and decide for yourself.
In reality, non-technical users will click on a first link that seems to be related to the question, and that's it.
Even with AI-based search (or q/a instead of search) I think the same will happen. There is and will be a huge reward for gaming the results, be they page links or RAG snippets that rank for a query. I've already seen many SEO shops advertising their strategies to keep the customers' business relevant in the chatbot area. As this approach becomes more prevalent, you can be sure many smart people will do many experiments to figure out how best to please the new algorithm.
In other words, AI-based search is an UX optimization, but doesn't address the core problem of how do you decide what content's the best, and do that in context of each user, and do that while maximizing the benefit for the user vs profit for the company doing this.
So we have two huge hurdles:
1. who will decide what the user wants[0] to see, and how are incentives for that entity aligned with the user's
2. how is that entity supposed to find the information needle in the haystack of slop that's 90%+ of current web?
[0] "wants" in a rational "give me the best possible information" meaning, not in "what keeps them addicted, their heart rate up, and what will drive engagement" meaning
> To paraphrase a recent conversation I had with a friend: "in the USA, can illegal immigrants vote?" has a single truthful answer ("no" obviously). But there are many places around the web saying other things (which is why my friend was confused).
This doesn’t have a single truthful answer. Some states don’t have voter ID laws, so the truth can depend on the state. In those no voter ID laws there’s not that much keeping someone from voting twice or more under different names, except significant moral qualms about subverting instead of preserving everyone else’s right to vote in a democratic republic. Someone can assume the name of a person from another country that could’ve plausibly come in illegally. Without a picture ID, they can’t claim you aren’t that persons.
Can an illegal immigrant vote? Yes, in states without voter ID laws, technically anyone can vote, even convicted terrorists. Should an illegal immigrant vote? No, they’re not supposed to be able to vote and there may be consequences if caught.
What purpose do a lack of voter ID laws serve except the obvious conclusion which is to enable cheating?
> With a search paradigm this wasn't an issue as much, because the answers were presented as "here's a bunch of websites that appear to deal with the question you asked". It was then up to the reader to decide which of those sites they wanted to visit, and therefore which viewpoints they got to see.
It is very similar. Google decides what to present to you on the front page. I'm sure there are metrics on how few people get past the front page. Heck, isn't this just Google Search's business model? Determining what you see (i.e. what is "true") via ads?
In much the same way that the Councils of Carthage chose to omit the acts of Paul and Thecla in the New Testament, all modern technology providers have some say in what is presented to the global information network, more or less manipulating what we all perceive to be true.
Recently advancements have just made this problem much more apparent to us. But look at history and see how few women priests there are in various Christian churches and you'll notice even a small omission can have broad impacts to society.
"Are all bachelors married?" vs "is it raining outside?"
And then Quine points out that the definitions of "bachelor" and "married" are themselves contingent on outside factors.
"Can illegal immigrants vote?", while close to being an analytic proposition, still depends on an empirical approach that can never be mediated by text, video, etc. All propositions are by necessity experiential. Nullius in verba!
So the truth is and has always been what happens when you get off your butt and go out and test the world.
This is not to say that we don't benefit from language. It makes for a great recipe. If you follow the instructions to bake a cake and you get what you expected you know that the recipe was true. The same goes for the laws of science, search engine results, and generative AI.
Unfortunately most people seem comfortable with the idea of a Ministry of Truth, at least here in Brazil and from what I read on the web, in many parts of the world also.
I think this presents an approach: move closer to the search engine model. In both academic writing an news journalism, factual claims try to cite their sources. And you can see this is an approach that some of the chat systems have directionally gone. We don't want to always pedantically cite a reference for every question. But maybe we should? At least at an architectural level and let the UX decide if it should be displayed.
> in the USA, can illegal immigrants vote
According Author X in Book Y which studied this topic in depth: foo answer
I don’t know who should get to decide “what is true” but I think we all agree that we won’t get anything remotely akin to truth so long as search is driven by for-profit interests trying to steer us towards their products first and our actual query (a distant) second.
> With a search paradigm this wasn't an issue as much, because the answers were presented as "here's a bunch of websites that appear to deal with the question you asked
I don't think this appropriately credits Google's power with regards to what you are seeing
For the record, in my local elections, we have plenty of people voting illegally.
"Is it legal" is very different from "Can they."
By "local," I mean municipal and below. I didn't mean "federal elections conducted in my locality." Election security kicks for state, federal, and some municipal elections.
Others are intentionally fraudulent (e.g. local corruption) or unintentionally broken (e.g. using Google Forms for a school-level public body, where people not legally qualified to vote might still do it, unaware they're committing a felony).
And "public body" has a specific meaning under my state law which extends the same laws as e.g. cutting for my state senate. That's bodies like local school boards, but not random school clubs.
That's the level where we have massive illegal voting where I live.
For any given statement, the answer up until a couple years ago was, "the speaker". Speakers get to decide what to say, but they're also responsible for what they say. But now with LLMs we have plausible text without a speaker.
I think we have a number of historical models for that. A relevant one is divination. If you bring your question to the haruspex, your answer is read out of the guts of a sacrificed animal. If the answer is wrong, who do you blame? The traditional answer is the gods, or perhaps nobody.
Bu we know now that fortune tellers are just selling answers while pretending to not be responsible for them. Which points us at one solution: anybody selling or presenting LLM output as meaningful is legally responsible for the quality of the product.
Unfortunately, another model is the modern corporation. Sometimes the people in a company intentionally lie. More often, statements are made by one person based on a vision or optimism or confusion or bullshit. Nobody set out to lie, but nobody really cared about the truth, at least not as much as everybody cared about making money.
So I'd agree that the government doesn't have much role in deciding The Truth. Similarly, the government shouldn't have much role in controlling what you eat. But in both cases, I think there's plenty of role for the government in ensuring that companies selling good food or good information have sound production and quality control measures to ensure that they are delivering what consumers are expecting.
> "in the USA, can illegal immigrants vote?" has a single truthful answer ("no" obviously)
legally no, practically- yes. In most states, you simply must attest that you are a citizen in order to register. In many states, non-citizens have been auto-registered to vote when attaining drivers licenses. Reddit is full of panicked immigrants concerned that they found themselves registered to vote, and worried about how that would affect their status.
Seems particularly to be a US based phenomenon. Unlike the more transparent manipulation seen under dictatorships... Where people generally recognize propaganda for what it is, even if they can’t openly challenge it—some in USA live within entirely new realities.
I genuinely think Kagi has led the way on this one. Simplicity is beautiful and effective, and Kagi has (IMHO) absolutely nailed it with their AI approach. It's one of those things that in hindsight seems obvious, which is a pretty good measure of how good an idea is IMHO.
Google could have done it and kind of tried, although they're AI sucks too much. I'm very surprised that OpenAI hasn't done this sooner as well. They're initial implementation of web search was sad. I don't mean to be super critical as I think generally OpenAI is very, very good at what they do, but they're initial browse the web was a giant hack that I would expect from an intern who isn't being given good guidance by their mentors.
Once mainstream engines start getting on par with Kagi, there's gonna be a massive wave of destruction and opportunity. I'm guessing there will be a lot of new pay walls popping up, and lots of access deals with the search engines. This will even further raise the barrier of entry for new search entrants, and will further fragment information access between the haves and have-nots.
I'm also cautiously optimistic though. We'll get there, but it's gonna be a bit shakey for a minute or two.
> I'm also cautiously optimistic though. We'll get there, but it's gonna be a bit shakey for a minute or two.
But I don't understand how all of these AI results (note I haven't used Kagi so I don't know if it's different) don't fundamentally and irretrievably break the economics of the web. The "old deal" if you will is that many publishers would put stuff out on the web for free, but then with the hope that they could monetize it (somehow, even just with something like AdSense ads) on the backend. This "deal" was already getting a lot worse over the past years as Google had done more and more to keep people from ever needing to click through in the first place. Sure, these AI results have citation results, but the click-through rates are probably abysmal.
Why would anyone ever publish stuff on the web for free unless it was just a hobby? There are a lot of high quality sites that need some return (quality creators need to eat) to be feasible, and those have to start going away. I mean, personally, for recipes I always start with ChatGPT now (I get just the recipe instead of "the history of the domestication of the tomato" that Google essentially forced on recipe sites for SEO competitive reasons), but why would any site now ever want to publish (or create) new high quality recipes?
Can someone please explain how the open web, at least the part of the web the requires some sort of viable funding model for creators, can survive this?
I gave Kagi a shot two weeks ago, and it instantly impressed me. I didn't realize how much search could be improved. It's a beautiful, helpful experience.
Could it be that Kagi benefits from being niche, though? Google search gets gamed because it’s the most popular and therefore gaming it gives the best return. I wonder if Kagi would have the same issues if it was the top dog.
> Will this fundamentally change how people find and access information? How do you create an experience so compelling that it replaces the current paradigm?
I think it's already compelling enough to replace the current paradigm. Search is pretty much dead to me. I have to end every search with "reddit" to get remotely useful results.
The concern I have with LLMs replacing search is that once it starts being monetized with ads or propaganda, it's going to be very dangerous. The context of results are scrubbed.
> The concern I have with LLMs replacing search is that once it starts being monetized with ads or propaganda, it's going to be very dangerous.
Not to mention that users consuming most content through a middle-man completely breaks most publishers business models. Traditional search is a mutually beneficial arrangement, but LLM search is parasitic.
Expect to see a lot more technical countermeasures and/or lawsuits against LLM search engines which regurgitate so much material that they effectively replace the need to visit the original publisher.
* Stackoverlow. Damaged through new owner but the idea lives.
* Reddit. Google tries to fuck it up with „Auto translation“?
* Gitlab or GitHub if something needs a bugfix.
The rest of the internet is either an entire ****show or pure gold pressed latinum but hardly navigatable thanks to monopolies like Google and Microsoft.
PS: ChatGPT already declines in answer because is source is Stackoverflow? And…well…these source are humans.
> I think it's already compelling enough to replace the current paradigm. Search is pretty much dead to me. I have to end every search with "reddit" to get remotely useful results.
I worry that there's a confusion here--and in these debates in general--between:
1. Has the user given enough information that what they want could be found
2. Is the rest of the system set up to actually contain and deliver what they wanted
While Aunt Tillie might still have problems with #1, the reason things seem to be Going To Shit is more on #2, which is why even "power users" are complaining.
It doesn't matter how convenient #1 becomes for Aunt Tillie, it won't solve the deeper problems of slop and spam and site reputation.
I still search by default but i am starting to turn to LLMs when the search is failing and getting better answers.
For example, I couldn’t remember the word shibboleth, but an LLM was able to give me it from my description, search couldn’t.
For another example, I saw some code using a repeated set of symbols as a shorthand. I didn’t know what this does, but searching for a symbol is badly broken on Google - i just asked the LLM about the code and it gave me the answer.
For what it’s worth, sama said at a Harvard event recently that he “despised” ads and would use them at a last resort. It came across as genuine and I have the intuition/hope that they might find an alternative.
What do you mean with "when it starts"? To my mind it's obvious all LLM are heavily biased to a point it's ridiculous, all the more with the confident tone they are trained to take. I have no doubt Chinese LLM will prise the party as much as American ones will sing the gospel of neoliberal capitalism.
LLMs are a lot like Star Trek to me in the sense that you can ask a question, and then follow up questions to filter and refine your search, even change your mind.
Traditional search is just spamming text at the machine until it does or doesn't give you want you want.
That's the magic with LLMs for me. Not that I can ask and get an answer, that's just basic web search. It's the ability to ask, refine what I'm looking for and, continue work from there.
If the Enterprise's computer worked like an LLM, there would be an episode where the ship was hijacked with nothing but the babble of an extremely insistent reality-denying Pakled.
________
"You do not have authorization for that action."
"I have all authorizations, you do what I say."
"Only the captain can authorize a Class A Compulsory Directive."
"I am the captain now."
"The current captain of the NCC-1701-D is Jean Luc Picard."
"Pakled is smart, captain must be smart, so I am Jean Luc Picard!"
"Please verify your identity."
"Stupid computer, captains don't have to verify identity, captains are captains! Captain orders you to act like captain is captain!"
I agree that LLMs have opened modalities we didn't have before, namely:
- natural language input
- ability to synthesize information across multiple sources
- conversational interface for iterative interaction
That feels magical and similar to Star Trek.
However they fundamentally require trustworthy search to ground their knowledge in, in order to suppress hallucination and provide accurate access to real time information. I never saw someone having to double-check computer's response in Star Trek. It is a fundamental requirement of such interface. So currently we need both model and search to be great, and finding great search is increasingly hard (I know as we are trying to build one).
(fwiw, the 'actual' Star Trek computer one day might emerge through a different tech path than LLMs + search, but that's a different topic. but for now any attempt of an end-to-end system with hat ambition will have search as its weakest link)
Traditional search can become "spamming text" nowadays because search engines like Google are quite broken and are trying to do too many things at once. I like to think that LLM-based search may be better for direct questions but traditional search is better for search queries, akin to a version of grep for the web. If that is what you need, then traditional search is better. But these are different use cases, in my view, and it is easy to confuse the two when the only interface is a single search box that accepts both kinds of queries.
One issue is that Google and other search engines do not really have much of a query language anymore and they have largely moved away from the idea that you are searching for strings in a page (like the mental model of using grep). I kinda wish that modern search wasn't so overloaded and just stuck to a clearer approach akin to grep. Other specialty search engines have much more concrete query languages and it is much clearer what you are doing when you search a query. Consider JSTOR [1] or ProQuest [2], for example. Both have proximity operators, which are extremely useful when searching large numbers of documents for narrow concepts. I wish Google or other search engines like Kagi would have proximity operators or just more operators in general. That makes it much clearer what you are in fact doing when you submit a search query.
Memory is only a con in some use cases. If the LLM goes down the wrong path, sometimes its impossible to get it to think differently without asking it to wipe its memory or starting a new session with a blank context.
> those visions, users simply asked questions and received reliable answers - nobody had to fact-check the answers ever.
It’s a fallacy then. If my mentor tells me something I fact check it. Why would a world exist where you don’t have to fact check? The vision doesn’t have fact checking because the product org never envisioned that outlier. A world where you don’t have to check facts, is dystopian. It means the end of curiosity and the end of “is that really true? There must be something better.”
You’re just reading into marketing and not fact checking the reality in a fact-check-free world.
If you can’t trust the result of a query how can you trust the check on that query which is itself a query? If no information is trustworthy how do you make progress?
Solving this problem will require us to stop using the entire web as a source of information. Anyone can write anything and put it up on the web, and LLMs have no way to distinguish truth from fantasy.
Limiting responses to curated information sources is the way forward. Encyclopedias, news outlets, research journals, and so on.
No, they're not infallible. But they're infinitely better than anonymous web sites.
You are quite right, and not only can anyone write anything, but you have a double whammy from the LLM which can further hallucinate from said information.
We had this long time before, it is called "books". It is also not very usable for niche topics, or for recent events (because curators need time to react).
> In those visions, users simply asked questions and received reliable answers - nobody had to fact-check the answers ever.
This also seems like a little ridiculous premise. Any confident statement about the real world is never fully reliable. If star trek were realistic the computer would have been wrong once in a while (preferably with dramatically disastrous consequences)—just as the humans it likely was built around are frequently wrong, even via consensus.
This feels like hyperbole to me. People can reasonably expect Wikipedia to have factual data even though it sometimes contains inaccuracies. Likewise if people are using ChatGPT for search it should be somewhat reliable.
If I'm asking ChatGPT to put an itinerary together for a trip (OpenAI's suggestion, not mine), my expectation is that places on that itinerary exist. I can forgive them being closed or even out of business but not wholly fabricated.
Without this level of reliability, how could this feature be useful?
Star Trek had tech so advanced that they accidentally created AGIs more than once. Presumably they didn't show the fact checking as it was done automatically by multiple, independent AGIs designed for the task with teams of top people monitoring and improving them.
I've been using ChatGPT for about 6 weeks as my go-to for small questions (when is sunset in sf today? list currencies that start with the letter P, convert this timestamp PDT to GMT, when is the end of Q1 2025?) and it's been great/99% accurate. If there was ever a "google killer" I think it's the ad free version of ChatGPT with better web search.
Google started off with just web search, but now you can get unit conversions and math and stuff. ChatGPT started in the other direction and is moving to envelope search. Not being directed to sites that also majority serve google ads is a double benefit. I'll gladly pay $20/30/mo for an ad free experience, particularly if it improves 2x in quality over the next year or two. It's starting to feel like a feature complete product already.
Remember the semantic web where people annotate search pages with data? I think the "meta" keywords tag was the closest that ever got to adoption.
Guess why it failed? It was largely used as a way to trick search engines. Same reason as your vision of the perfectly honest and correct search engine and look chatbot will never be perfect. It's because people lie to search engines to spam and get traffic they don't deserve. The whole history of search is Google and others dealing with spam. Same goes for email. Google largely defeating spam made them the kings of the email world.
Everyone will need their own personal spam filter for the world for everything ones the artificial super intelligences fill the whole world with spam, scams and just plain old social engineering propaganda because we will be like helpless four year old children in the world of AI super intelligence without our AI parents to look out for us.
Your vision of the god system determining what is truth is like saying there will be a single source of truth for what is and is not a spam email. Not going to scale and not going to be perfect, but good enough with AI and technology. Really hope there's an opt-out though since Google had memoryholed most of the Internet.
Google created a printing money machine. OpenAI is absolutely trying to create one too. They aren't trying to "fix the web". Search is a proxy to get into people's wallet. OpenAI might directly get into it buying agents. The problem of miss aligned incentives on the search space isn't a technological problem. A new technology might solve it as a side effect, but it is unlikely IMO.
Your points are good but I wonder if you’re wishing for an ideal that has never existed:
> actually surfacing the content people want to see, not what intermediaries want them to see
Requires two assumptions, 1) the content people want to see actually exists, 2) people know what it is they want to see. Most content is only created in the first place because somebody wants another person to see it, and people need to be exposed to a range of content before having an idea about what else they might want to see. Most of the time what people want to see is… what other people are seeing. Look at music for example.
Great find on the knowledge navigator, I had never seen it but I was a toddler when it was released haha.
It's interesting how prescient it was, but I'm more struck wondering--would anyone in 1987 have predicted it would take 40+ years to achieve this? Obviously this was speculative at the time but I know history is rife with examples of AI experts since the 60s proclaiming AGI was only a few years away
Is this time really different? There's certainly been a huge jump in capabilities in just a few years but given the long history of overoptimistic predictions I'm not confident
> It's interesting how prescient it was, but I'm more struck wondering--would anyone in 1987 have predicted it would take 40+ years to achieve this? Obviously this was speculative at the time but I know history is rife with examples of AI experts since the 60s proclaiming AGI was only a few years away
40+ makes it sound like you think it will ever be achieved. I'm not convinced.
Thinking about incentive alignment, non ad-based search would be better than ad-based, but there'd still be misalignment due to the problem of self-promotion. Consider Twitter for example. Writing viral tweets isn't about making money (at least until recently), but the content is even worse than SEO spam. There is also the other side of the problem that our monkey brains don't want content that's good for us in the long run. I would _love_ to see (or make) progress in solving this, but this problem is really hard. I thought about it a lot, and can't see an angle of attack.
Don’t you see it coming ? Contrary to google search the engine knows you and will put personalized ads on synthesized answers. Even generate a small video ad of a product or service, tailored to you on the spot.
That, is the future. This is the endgame. It’s what made Google rich, why not use the same formula on steroids.
As someone who has been using google for search basically constantly since 1995, I've switched probably 90%+ of what normally would have been google searches over to Perplexity (which gives me references to web pages alongside answers to my questions, to review source materials) and ChatGPT (for more just answers I can verify without source). The remaining searches have gone to Kagi.
On the one hand this has got to be hurting Google search ad revenue. On the other hand I don't know if I ever clicked an advertised link. On the other other hand, not having to wade through SEO results has been so nice.
I tend to be more pessimistic about the incentives getting fixed than others are. I also think the situation is more complex than some of the people replying to you.
(1) Search is already heavily AI driven, and Google is clearly going in that direction. Gemini is currently separate, but they'll blend it in with search over time, and no doubt search already uses LLM for some tasks under the hood. So ChatGPT search is an evolution on the current model rather than a big step in a new direction. The main benefit is you can ask the search questions to refine or followup.
(2) Aside from the economic incentives faced by search engines, there is the fact that algorithms are tuned toward a central tendency. The central tendency of question askers will always be much less informed than the most informed extreme. Google was much better when the average user was technical. The need to capture mobile searches is one force that made it return on average worse results. Similarly if Kagi has a quality advantage now, we need to be realistic about how much of that quality is driven by its users being more technical.
(3) I think micropayment schemes have generally asked several orders of magnitude more for a page view than users are willing to pay. As long as content creators value their content much more highly than consumers do, they'll stick with advertising which lets them overcharge and gives consumers less of an option to say no to the content.
> Combining two broken systems - compromised search engines and unreliable LLMs - seems unlikely to yield that vision
Counterpoint: with a chain-of-thought process running atop search, you can potentially avoid much of the meta-search / epistemic hygiene work currently required. If your “search” verb actually reads the top-100 results, runs analyses for a suite of cognitive biases such as partisanship, and gives you error bars / warnings on claims that are uncertain, the quality could be dramatically improved.
There are already custom retrieval/evaluation systems doing this, it’s only a matter of a year or two before it’s commoditized.
The concern is around OpenAI monetization, do they eventually start offering paid ads? This could be fine if the unpaid results are great, a big part of why the web is perceived to be declining is link-spam that Google doesn’t count as an ad.
My prediction would be that there is a more subtle monetization channel; companies that can afford to RAG their products well and share those indexes with AI search providers will get better results. RAG APIs will be the new SEO.
This product vision stupidity is present in every one of googles products. Maps has feed for some reason. The search in it for what I want is horrendous. There is no coherence between time(now) and what a person can do around the current location.
Navigation is the only thing that works but wayz was way better at that and the only reason they killed(cough bought it) was to get the eyeballs to look at feed.
The most rational incentive would seem to be a kind of toll extracted on the value of all you create where search contributed to that creation. So like every month, you get an invoice from the great search company in the cloud where somehow it’s objectively assessed how much search contributed to the value you created for the world that month - and then you pay that amount and all remains right with the world until the end of time.
In that way, search captures value directly on its core function: efficiently facilitating the creation and dispersal of knowledge.
This may end up occurring, but based on how unlikely it sounds, it’s my reflection that the web search that we have today is simply a small component of what will eventually become a system that closely approximates the above: basically a kind of global intelligent interconnected agent for the advancement of humanity.
Rather than the great unbundling, this will be the great rebundling … of many diverse functions into a single interface seamlessly.
I did a quick test "Best air conditioner by value", comparing Bing and ChatGPT search, and the rankings on the first two pages of Bing vs sources on ChatGPT were about half the same. It's interesting that there is some deviation where ChatGPT isn't blindly trusting the rankings Bing provides and is going deeper into the results to find something more relevant. It's a good improvement over products like Arc Search.
Seeing search enter the space is something that I feel has been seriously needed, as I've slowly replaced Google with ChatGPT. I want quick, terse answers sometimes, not a long conversation, and have hundreds of tiny chats. But there's something scary seeing the results laid out the way they are, which leads me to believe they may be closer to experimenting with ad-based business models, which I could never go back to.
I use the “web search” bot on Poe.com for general questions these days, that I previously would have typed into Google (Google’s AI results are sometimes helpful though). It is better than GPT (haven’t tried TFA yet though), because it actually cites websites that it gets answers from, so you can have those and also verify that you aren’t getting a hallucination.
Besides Poe's Web Search, the other search engine I use, for news but also for points of view, deep dive type blog type content, is Twitter. Believe it or not. Google search is so compromised today with the censorship (of all kinds, not just the politically motivated), not to mention Twitter is just more timely, that you miss HUGE parts of the internet - and the world - if you rely on Google for your news or these other things.
The only time I prefer google is when I need to find a pointer/link I already know exists or should exist, or to search reddit or HN.
I think to solve this problem we need to take a step back and see how a human would solve the problem (if he had all the necessary information).
One thing is clear, in the vast majority of cases we don't have a single truth, but answers of different levels of trustworthiness: the law, the government, books, Wikipedia, websites, and so on.
A human would proceed differently depending on the context and the source of the information. For example, legal questions are best answered by the law or government agencies, then by reputable law firms. Opinions of even highly respected legal professionals are clearly less reliable than government law itself and are likely to be a point of contention/litigation.
Questions about other facts, such as the diameter of the earth or the population of a country, are best answered by recent data from books, official statistics and Wikipedia. And so on and so forth.
If we are not sure what the correct answer is, a human would give more information about the context, the sources, and the doubts. There are obviously questions that cannot be answered immediately! (if it's to easy to find the truth, we would not need a legal system or even science!) So no machine and no amount of computation can reliably answer all questions. Web search does not answer a question. It's just trying to surface relevant websites to a bunch of keywords. The answering part is left as an exercise for the user ;)
So an AI search with a pretense of understanding human languages makes the task incredibly harder. To really give a human-quality answer, the AI not only needs to understand the context, but it should also be able to reason, have common sense, and be a bit self-aware (I'm not sure...). All this is beyond the capabilities of the current generation of AI. Therefore, my conclusion is that the "search" or better said the "answer" cannot be solved by LLM, no matter how much they fine-tune and tweak it.
But we humans are adaptable. We will find our way around and accept things as they are. Until next time.
Interesting, I hadn’t see the Knowledge Navigator before. I would argue that we’re very close to the capabilities shown in that video.
Isn’t this already that? A new business model? Something like OpenAI’s search or Perplexity can run on its own index and not be influenced by Google’s ranking, ads, etc.
In areas where there is a simple objective truth, like finding the offset for the wheels on a 2008 BMW M3, we have had this capability for some time with Perplexity. The LLMs successfully cuts through the sea of SEO/SEM and forum nonsense and delivers the answer.
In areas where the truth is more subjective, like what is the best biscuit restaurant in downtown Nashville, the system could easily learn your preferences and deliver info suited to your biases.
In areas where “the science” is debated, the LLM can show both sides.
> Path forward requires solving the core challenge: actually surfacing the content people want to see, not what intermiediaries want them to see
But this will never happen with mainstream search imo. It is not a technical problem but a human one. As long as there is a human in control of what gets surfaced, it is only a matter of time until you revert to tampered search. Humans are not robots. They have emotions and can be swayed with or without their awareness. And this is a form of power for the swayer as much as oil or water are.
The idea that you can have an AI system provide factual and reliable answers to human centric questions is as real as Star Trek itself.
You will never remove the human factor from AI
Your hope might be that a technical solution is found for a human problem but that is unlikely.
Regarding incentives - with Perplexity, ChatGPT search et al. skinning web content - where does it leave the incentive to publish good, original web content?
The only incentivised publishing today is in social media silos, where it is primarily engagement bait. It's the new SEO.
I was thinking about this myself, so I went to another search engine (Bing) which I never use otherwise, and jumped right into their "Copilot" search via the top navbar.
Man, it was pretty incredible!
I asked a lot of questions about myself (whom I know best, of course) and first of all, it answered super quickly to all my queries letting me drill in further. After reading through its brief, on-point answers and the sources it provided, I'm just shocked at how well it worked while giving me the feeling that yes, it can potentially – fundamentally change things. There are problems to solve here, but to me it seems that if this is where we're at today, yes in the future it has the potential to change things to some extent for sure!
I ask myself why did we accept pre 2010s answers, maybe because the media institutions had accumulated enough trust ? I feel this unavoidable.. nothing is true unless there's a threshold of quality/verification in place across the system.
The media was lying all the time. It didn't start lying just after advent of social media. Just look at sitcoms like Yes Minister. The reason we now consider media to lie is because their lies are exposed, documented, tracked and refreshed to remind users not to trust them blindly.
I'm not sure this is any different than any time before.
Before we had the internet, how did you answer questions? You either looked it up in a book, where you then had to make a judgment on whether you trusted the book, or you asked a trusted person, who could give you confidently wrong answers (your parents weren't right every time, were they? :) ).
I think the main difference here is that now anyone can publish, whereas before to make a book exist required the buy in of multiple people (of course, there were always leaflets).
The main difference now is distribution. But you still, as a consumer of information, have to vet your sources.
I couldn't agree more. Neither LLMs or search engines yield reliable answers due to the ML model and misaligned incentives respectively. Combining the two is a shortsighted solution that doesn't fix anything on a fundamental level. On top of that we have the feedback loop. LLMs are used to search the web and write the web, so they'll just end up reading their own (unreliable) output over time.
What we need is an easier way to verify sources and their trustworthiness. I don't want an answer according to SEO spam. I want to form my own opinion based on a range of trustworthy sources or opinions of people I trust.
I saw products in my search today and reverted back to my main concern on how AI is going to get paid for —> varying levels of advertisement, product placement etc.
We are currently in the growth phase of VC funded products where everything is almost free or highly subsidized (save chats sub) - i am not looking forward to when quality drops and revenue is the driving function.
We all have to pay for these models somehow - either VC lose their equity stakes and it goes to zero (some will) or ads will fill in where subs don’t. Political ads in AI is going to wreak havoc or undermine the remainder of the product.
I think the 'Age of PageRank' could be revived. A few years ago I had this idea about a decentralized, distributed, independent, public, universal, dynamic and searchable directory of websites; I wrote a toy specification document for it [1] - which I had forgotten until reading this discussion - and if I'm not mistaken, implementing it could be a few days project for a regular dev.
I do love how all the comments on the top comment of every HN post just consists of deeper and deeper levels of inane bickering ahaha. It's like a template.
I am excited for a future where I search for some info and don't end up sifting through ads and getting distracted by some tangential clickbait article.
Fundamentally it feels like that cant happen though because there is no money in it, but a reality where my phone is an all knowing voice I can reliably get info from instead of a distraction machine would be awesome.
I do "no screen" days sometimes and tried to do one using chatGPT voice mode so I could look things up without staring at a screen. It was miles from replacing search, but I would adopt it in a second if it could.
Even with perfect knowledge right now, there’s no guarantee that knowledge will remain relevant when it reaches another person at the fastest speed knowledge is able to travel. A reasonable answer on one side of the universe could be seen as nonsensical on the other side - for instance, the belief that we might one day populate a planet which no longer exists.
As soon as you leave the local reference frame (the area in a system from which observable events can realistically be considered happening “right now”), fact checking is indeed required.
The whole basis for information and data is totally corrupted and we are not even allowed i.e., enabled to talk about that, let alone the corrupted nature of the information and data, largely because entrenched tyrannical people and their mindsets are in total control of the mainstream and have even made significant headway in shattering the dissident opposition. The consumption side incentives are effectively irrelevant when what is consumed is basically worthless. What’s the value of lies?
I was thinking of the direction we are going and even wanted to write up a blog about it. IMO the best way forward would be if AI can have some logical thoughts independent of human biases but that can only happen if AI can reason unlike our current LLMs that just regurgitate historical data.
> Path forward requires solving the core challenge: actually surfacing the content people want to see, not what intermiediaries want them to see
This is flawed thinking to get to the conclusion of “reliable answers”. What people want to see and the truth are not overlapping.
Consider the answers for something like “how many animals were on Noah’s Ark” and “did Jesus turn water into wine” for examples that cannot be solved by trying get advertisers out of the loop.
Yes the incentives will need to change. I think it’s also going to be a bigger question than just software. What do we do in general about those that control capital or distribution channels and end up rent seeking? How do others thrive in that environment?
In the short term, I wonder what happens to a lot of the other startups in the AI search space - companies like Perplexity or Glean, for example.
You're absolutely right: today's search engines and language models are limited by misaligned incentives. Ad-driven search engines prioritize content that is optimized for algorithms and ad revenue, often at the expense of depth and quality, while language models inherit these biases, which compromises their reliability.
I suspect the future old man thing I will do is use a search engine when my kids ask plain English questions to a bot. I’ll tell them “but LLMs hallucinate, I don’t trust those things!” and they’ll just laugh off my old man ways. The irony will be that 95% search results are AI generated anyway by that point.
Chat isn't needed to provide reliable answers. Google used to do this over a decade ago. What Star Trek didn't foresee was vested interests in the SEO space, governments, political special interest groups, and the owners of the search engines themselves had far too much incentive to bork and bias results in their favor. Google is an utter shit show. More than half the time it won't find the most basic search query for me. Anything older than a couple years, good luck. I'm sure it's just decision after decision that piled up, each seemingly minor in isolation but over the years has made these engines nearly worthless except for a particular window of cases.
But it will radically increase the energy requires to deliver search results, which means more datacenters and more potential for the monitization of "premium" search services. Getting people to pay for something they today get for free always looks good on paper.
Had an experience yesterday where a simple mdn doc existed but chatgpt gave the reverse of reality on that doc. Wasted about an hr of my time, but taught me these things will hallucinate even the simplest of stuff at times. Not sure how we even fix that.
there still has to be some ranking feature for the backend search database to return the top n results to the LLM. So pagerank isn't over, it's just going to move to a supporting role, and probably modified as the SEO arms race continues.
Will ChatGPT (and other products like it) find a niche use case for what Google search covers? Yes.
Will it replace Google in the mass market? No. Why? Power. I don't mean how good the product is. I mean the literaly electricity.
There are key metrics that Google doesn't disclose as part of its financials. These include thing slike the RPM (Revenue per Thousand Searches) but it also must include something like the cost of running a thousand searches when you amortize everything involved. All the indexing, the software development and so on. That will get reduced to a certain amount of CPU time and storage.
If I had to guess, I would guess that ChatGPT uses orders of magnitude more CPU power and electricity than the average Google search.
Imagine trying to serve 10-50M+ (just guessing) ChatGPT searches every secondj. What kind of computing infrastructure would that take? How much would it cost? How would you monetize it?
Another way to put it is simply that ChatGPT search is built on top of existing search engines. The best case scenario is that it cherry picks the best from all available search engines. It can’t totally supersede all search engines.
The computer in Star Trek was not owned by a for-profit company. Ir has different incentives. Also, i think its knowledge base Is curated (a kind of starfleet library, with no random entries for a intergalactic internet).
People in time will learn that using a GPT for search especially for a question that involves research where you had to do MANY searches for the answer... now is provided via one query.
Making things quicker and easier always wins in tech and in life.
The biggest question is, can this bring back the behaviour of search engines from long ago? It's significantly difficult to find old posts, blogs or forums with relevant information compared to 10-15 years ago.
Star Trek is the vision of a better world we all need in so many aspects. It’s crazy nowadays, cause we seemed to got further away instead of closer to those visions.
At least less people are hungry.
I think the promise of search was to be able to do confidence-ranked "grep" on the internet. Unfortunately we departed from this, but it's what I desperately want.
What if I am looking for a medical page or a technical page or whatever else where I need to see, read, experience actual content and not some AI summary?
We have a faulty information network to begin with, and have for millenia. There's no such thing as "reliable" answers in a world full of unreliable humans.
This will probably be extremely radical and controversial in this contemporary world.
We need to stop adopting this subscription model society mentality and retake _our_ internet. Internet culture was at one point about sharing and creating, simply for the sake of it. We tinker'd and created in our free time, because we liked it and wanted to share with the world. There was something novel to this.
We are hackers, we only care about learning and exploring. If you want to fix a broken system, look to the generations of old, they didn't create and share simply to make money, they did it because they loved the idea of a open and free information super highway, a place where we could share thoughts, ideas and information at the touch of a few keystrokes.
We _have_ to hold on to this ethos, or we will lose what ever little is left of this idea.
I see things like kagi and is instantly met with some new service, locked behind a paywall, promising lush green fields of bliss. This is part of the problem.
(not saying kagi is a bad service)
I see a normalized stigma around people who value privacy, and as a result is being locked out, behind the excuse of "mAliCiOuS" activity.
I see monstrous giants getting away with undermining net neutrality and well established protocols for their own benefit.
I implore you all, young and old, re(connect) to the hacker ethos, and fight for a free and open internet. Make your very existence a act of rebellion.
gpt + search have been married long ago, for example, Phind. The problem that recently emerged, it tends to find long articles written by GPT, and lacking any useful information.
I hope we see more evolution of options before it does. Hard to articulate this without it becoming political, but I've seen countless examples both personally and from others of ChatGPT refusing to give answers not in keeping with what I'd term "shitlib ethics". People seem unwilling to accept that a system that talks like a person may surface things they don't like. Unless and until an LLM will return results from both Mother Jones and Stormfront, I'm not especially interested in using one in lieu of a search engine.
To put this differently, I'm not any more interested in seeing stormfront articles from an LLM than I am from google, but I trust neither to make a value judgement about which is "good" versus "bad" information. And sometimes I want to read an opinion, sometimes I want to find some obscure forum post on a topic rather than the robot telling me no "reliable sources" are available.
Basically I want a model that is aligned to do exactly what I say, no more and no less, just like a computer should. Not a model that's aligned to the "values" of some random SV tech bro. Palmer Luckey had a take on the ethics of defense companies a while back. He noted that SV CEOs should not be the ones indirectly deciding US foreign policy by doing or not doing business. I think similar logic applies here: those same SV CEOs should not be deciding what information is and is not acceptable. Google was bad enough in this respect - c.f. suppressing Trump on Rogan recently - but OpenAI could be much worse in this respect because the abstraction between information and consumer is much more significant.
> Basically I want a model that is aligned to do exactly what I say
This is a bit like asking for news that’s not biased.
A model has to make choices (or however one might want to describe that without anthropomorphizing the big pile of statistics) to produce a response. For many of these, there’s no such thing as a “correct” choice. You can do a completely random choice, but the results from that tend not to be great. That’s where RLHF comes in, for example: train the model so that its choices are aligned with certain user expectations, societal norms, etc.
The closest thing you could get to what you’re asking for is a model that’s trained with your particular biases - basically, you’d be the H in RLHF.
I buy that there's bias here, but I'm not sure how much of it is activist bias. To take your example, if a typical user searches for "is ___ a Nazi", seeing Stormfront links above the fold in the results/summary is going to likely bother them more than seeing Mother Jones links. If bothered by perceived promotion of Stormfront, they'll judge the search product and engage less or take their clicks elsewhere, so it behooves the search company to bias towards Mother Jones (assuming a simplified either-or model). This is a similar phenomenon to advertisers blacklisting pornographic content because advertisers' clients don't want their brands tainted by appearing next to things advertisers' clients' clients ethically judge.
That's market-induced bias--which isn't ethically better/worse than activist bias, just qualitatively different.
In the AI/search space, I think activist bias is likely more than zero, but as a product gets more and more popular (and big decisions about how it behaves/where it's sold become less subject to the whims of individual leaders) activist bias shrinks in proportion to market-motivated bias.
Kagi also drinks the koolaid, namely the knowledge navigator agent bullshite.
"The search will be personal and contextual and excitingly so!"
---
Brrrr... someone is hell-bent on the extermination of the last aspects of humanity.
Holy crap, this will be next armageddon, because people will further alienate themselves from other people and create layers of layers of unpenetrable personal bubbles around themselves.
Kagi does the same what google does, just in a different packaging. And these predictions, bleh, copycats and shills in a nicer package.
We will get there when ppl move past capitalism and socialism.
Like an ant colony pushing into one direction.
It will happen, but we need few more global dying events / resets.
I believe human race can get there but not in current form and state of mind.
> In those visions, users simply asked questions and received reliable answers - nobody had to fact-check the answers ever.
I mean, Star Trek is a fictional science-fantasy world so it's natural that tech works without a hitch. It's not clear how we get there from where we are now.
"Yes i'd like to help you with your homework but on a side note i'll recommend you this tablet, science shows people who's done this purchase are much happier people, also remember the financial plutocracy is watching out for you, they are friends of the working people, and the war machine makes the world much safer for everyone so please do go ahead and vote for either of the two pro world peace forever parties, no other ideologies or parties are safe"
> Legacy ad-based search has devolved into a wasteland of misaligned incentives, conflict of interest and content farms optimized for ads and algos instead of humans.
> Path forward requires solving the core challenge: actually surfacing the content people want to see, not what intermediaries want them to see
These traps and patterns are not inevitable. They happen by choice. If you're actively polluting the world with AI generated drivel or SEO garbage, you're working against humanity, and you're sacrificing the gift of knowing right from wrong, abandoning life as a human to live as some insectoid automaton that's mind controlled by "business" pheromones. We are all working together every day to produce the greatest art project in the universe, the most complex society of life known to exist. Our selfish choices will tarnish the painting or create dissonance in the music accordingly.
The problem will be fixed only with culture at an individual level, especially as technology enables individuals to make more of an impact. It starts with voting against Trump next week, rejecting the biggest undue handout to a failed grifter who has no respect for law, order, or anyone other than himself.
It is so absurd having to spend so much energy on "what happened on the football match yesterday" just because the internet is a wasteland full of ads.
This is probably Google's Altavista moment, by making their results crappier by the year in search of Ad dollars everyone has felt that there is a potential for search to be better and once that becomes available they'll be in a continious game of catch-up.
Yes, Google has their own AI divisions, tons of money and SEO is to blame for part of their crappiness. But they've also _explicitly_ focused on ad-dollars over algorithmic purity if one is to believe the reports of their internal politics and if those are true they have probably lost a ton of people who they'd need right now to turn the ship around quickly.
At some point it seems like Google switch to ML-based search instead of index based search. You can search for very specific combinations of lyrics and scenes: "eyes on me pineapple bucket of water house of cards chess time loop" and you won't surface a link to the music video featuring all of those things (https://www.youtube.com/watch?v=AlzgDVLtU6g), you'll just get really generic results of the average of your query.
Has google completely stopped working for anyone else?
I can still search things, i get results but, they're an ordered list of popular places the engine is directing me to. Some kind of filtering is occurring on nearly every search i make that's making the results feel entirely useless.
Image search stopped working sometime ago and now it just runs an AI filter on whatever image you search for, tells you there's a man in the picture and gives up.
Youtube recommendations is always hundreds of videos i've watched already, with maybe 1-2 recommendations to new channels when i know there's millions of content creators out there struggling who it will never introduce me to. What happened to the rabbit holes of crazy youtube stuff you could go down?
This product is a shell of its old self, why did it stop working?
FWIW, ChatGPT Search didn't surface the video either with that query
> Based on the elements you’ve described—eyes, a pineapple, a bucket of water, a house of cards, chess, and a time loop—it’s challenging to identify a single music video that encompasses all these features.
I think this is a separate issue although it also exists.
What the parent is referring to is favoring annoying ad-filled garbage over an equally relevant but straightforward result.
The hidden variable is that ad-riddled spam sites also invest in SEO, which is why they rank higher. I am not aware of any evidence that Google is using number of Google ads as a ranking factor directly. But I would push back and say that “SEO” is something Google should be doing, not websites, and a properly optimized search engine would be penalizing obvious garbage.
Yes and no. Yes for the quality of search results. Google algorithm and user experience was simply better than AltaVista's, but Google had another advantage. It used a cluster of cheap consumer-grade hardware (PCs) as its backend, rather than the expensive servers AltaVista used, in fact, AltaVista started off as a way for DEC to show off its hardware.
As a result Google was not only better than its competitors at searching, it was also cheaper and scaled better, and this architecture became the new standard.
It is the opposite for these AI-based systems. AI is expensive, it uses a lot of energy, and the hardware is definitely not consumer-grade.
What it means is that barring a significant breakthrough, cheap ChatGPT-like services are simply unsustainable. Google don't have to do anything, it will collapse on its own. Ironically, probably in the same way that Google result became crappier by the year, but on fast forward.
> What it means is that barring a significant breakthrough, cheap ChatGPT-like services are simply unsustainable.
This is the basic premise of Ed Zitron's article https://www.wheresyoured.at/to-serve-altman/ and others written around the same time. A lot of what he's written seems to be coming to pass, e.g. unfathomably large investments (~$100 billion) by tech giants in OpenAI and other AI startups to keep them going. That does seem unsustainable.
Can anyone make a counterpoint to this claim? I'd be very interested to hear what a viable path to profitability looks like for OpenAI.
In essence there is no “objective” algorithm. It’s a mish mash of random black box patterns that we hope delivers the right answer. It probably will most of the time but it will be gamed hard and it’s going to be hard to undo changes in gamified models.
You’re also giving Google feedback when you click a search result link. Presumably that should be a huge signal for measuring search quality.
Heck, Google even promoted the `ping`[0] anchor attribute feature so they can log what link you click without slowing you down. (Firefox doesn’t support ping, which means when Firefox users click on a Google search result link they’re sent to an internal google.com URL first and then redirected for logging purposes)
I have no doubt that Google’s search team is optimizing for the best results. The problem is their ads team is optimizing for revenue. You can’t optimize for two things at the same time without compromising (the optimum is the Pareto frontier).
Is it optimising for all users? And assuming people thumbs up the correct info. I wonder what accuracy percentage we are looking at there. ChatGPTs responses are so confident when wrong I fear people will just give it a thumbs up when its wrong. (This is if how I understand the feature you mention is working)
One thing I will always admire Apple for is being willing to essentially kill the iPod with the iPhone in 2007. No other company I've seen has been willing to kill their cash cow so thoroughly and so profitably.
Google is entirely to blame. It would be trivial for them to train a model to rank sites on a scale of SEO garbage to nobel laureate essay, then filter out the bottom 50%.
To be honest, this looks more like OpenAI playing catchup to what Google’s done since 2023. The fact is that ChatGPT, which people thought would kill Search, is falling behind Google in some ways.
I think it’s great that competition is driving both companies to improve, but I’m not seeing anything about this that screams “Google-killer”
> But they've also _explicitly_ focused on ad-dollars over algorithmic purity
I think they prioritized large and fast incomes over long, steady incomes.
Also, there isn't just money that taints Google search results. It's also censorship and political motivation, which is mostly obvious in image search.
There's this tech pattern of letting the cash-cow stagnate and deteriorate while focusing on high risk moonshots.
This especially happens after they dominate the market.
Take for example IE6, Intel, Facebook, IBM, and now Google.
They have everything they need to keep things from going off the rails, management however has a tendency to delusionally assume their ship is so unsinkable that they're not even manning their stations.
It becomes Clayton Christensenesque - they're dismissive of the competition as not real threats and don't realize their cash-cow is running on only fumes and inertia until its too late.
I think it’s what is the natural consequence of handing over management to MBA types. They can’t create new things. I don’t mean that they should be programmers, engineers or whatever they’re working with. I mean that they can only exact value from something that already exists. Which is always going to be a short term strategy because the only way to do that is to make “the it” worse.
I’m not sure Facebook fits in considering they at least managed to get some other products along the way, and may get more.
I certainly don’t think Google fits the bill. Google is failing because they let their cash cow ruin everything else, not because they let it stagnate while they chased the next moonshot. Google Cloud could have easily been competitive with AWS and Azure in European Enterprise, but it’s not even considered an option because Google Advertising wouldn’t let it exist without data harvesting. Google had Office365 long before Microsoft took every organisation online. But Google failed to sell it because… well…
It’s very typical MBA though. Google has killed profitable products because those products weren’t growing enough. A silly metric really, but one which isn’t surprising when your CEO is a former McKinsey.
It couldn’t happen to a nicer company though, and at least it won’t kill people unlike Boeing.
> There's this tech pattern of letting the cash-cow stagnate and deteriorate while focusing on high risk moonshots.
It is interesting to go back a decade and read the articles about Google's moonshots from then. There was even more "we're building the future!" hype than what Elon Musk or Sam Altman are currently pushing.
More competition is good, but I’m pretty sure the end result will just be, at best, a duopoly of products equally ruined by advertising for their “free” tier, and somewhat less ruined for the “premium” tier.
I'm wondering if they've seen the writing on the wall for a long time, and if diversification is going to save the day. I don't know that I've ever clicked on an ad in Google search results, but I happily pay them for a family account for Youtube and 2TB of drive storage, plus, you know, a flagship phone every couple years. :-)
All businesses seek to diversify to reduce risk. That doesn't mean they have any reason to believe a cataclysmic event for search is more likely now vs before. ChatGPT to search feels like a similar situation of macos is to windows. It may take market share but the ecosystem around Windows keeps it by far in the lead in market share. Also Microsoft diversifying away from windows enabled that market share loss to not really matter all that much.
Google, even with all the trillions in additional productivity it has added to the world, has left the world at a net-negative. We can't even quantify it, and every person will tell you a different way in which it has impacted the world negatively.
E.g. for me, how much Google (and silicon valley in general) have enabled twisted ideologies to flourish. All in search of ad-dollars by virtue of eyeballs on screens, at the detriment of everything.
Anyone have some good links/citations on the consumer surplus of Google? How much value do we get as users?
Considering the value of time, past consumer surplus is especially valuable now.
Sure, there are systematic flaws causing SEO to ruin the information provided: but it isn't clear what Google can do to fight the emergent system.
I'm not sure that Bing/DDG are any better.
I use search (DDG web, Google/Apple maps, YouTube) all the time and I am regularly given results that are extremely valuable to me (and mostly only directly cost me a small amount of my time some of my time e.g. YouTube adverts). Blaming SEO on Google seems thoughtless to me. Google appears to be the victims of human cybersystems as much as we are.
I'm seeing the trillions in additional productivity argument more often here, especially when trying to defend their monopolistic tendencies. What I've never seen are raw numbers backing this.
I was just wondering what has come out of silicon valley since say 2003 that has been a net positive for humanity. Just because something is profitable doesn't mean it's progress.
I know this is a predominantly white, upper middle class site (read: progressive), and people from that demo sure love to point out that "you can't say the leg is broken just because the foot is dangling at 90 degrees, you aren't a doctor!" ..... but at some point you surely have to start to trust your own eyes and use critical thinking, right??
It's not a conspiracy theory Google ruined their search to serve more ads. It's so obvious and happening for such a long period of time, anyone above certain age and with decent memory can see it. You don't have to "believe the reports", you don't even have to be smart, just use your eyes and your memory.
If the current iteration of search engines are producing garbage results (due to an influx of garbage + SEO gaming their ranking systems) and LLMs are producing inaccurate results without any clear method proposed to correct them, why would combining the two systems not also produce garbage?
The problem I see with search is that the input is deeply hostile to what the consumers of search want. If the LLM's are particularly tuned to try and filter out that hostility, maybe I can see this going somewhere, but I suspect that just starts another arms race that the garbage producers are likely to win.
Search engines tend to produce neutral garbage, not harmful garbage (i.e. small tidbits of data between an ocean of SEO fluff, rather than completely incorrect facts). LLMs tend to be inaccurate because in an absence of knowledge given by the user, it will sometimes make up knowledge. It's plausible to imagine that they will cover each other's weaknesses: the search engine produces an ocean of mostly-useless data, and the LLM can find the small amount of useful data and interpret that into an answer to your question.
The problem I see with this "cover for each other" theory is that as it stands having a good search engine is a prerequisite to having good outputs from RAG. If your search engine doesn't turn up something useful in the top 10 (which most search engines currently don't for many types of queries) then your llm will just be summarizing the garbage that was turned up.
Currently I do find that Perplexity works substantially better then Google for finding what I need, but it remains to be seen if they're able to stay useful as a larger and larger portion of online content just AI generated garbage.
> Search engines tend to produce neutral garbage, not harmful garbage (i.e. small tidbits of data between an ocean of SEO fluff, rather than completely incorrect facts)
Wasn't google AI surfacing results about making pizza with glue and eating rocks? how is that not harmful garbage?
Garbage-ness of search results is not binary, the right question is: can LLMs improve the quality of search results? But sure, it won't end the cat and mouse game.
I think that's the right broad question. Though LLMs properties mean that for some number of cases they will either make the results worse, or more confidently present wrong answers. This prompts the question: what do we mean by "quality" of results? Since the way current LLM interfaces tend to present results is quite different from traditional search.
The question is what is the business model and who pays for it, that determines how much advertising you’re getting. It is not clear if OpenAI could compete in Ad-supported search. So maybe OpenAI is trying to do the basic research, outcompete the Bing research group at Microsoft and then serve as an engine for Bing. Alternatively they could be just improving the ability of LLMs to do search, targeting future uses in agentic applications.
There is no way to SEO the entire corpus of human knowledge. ChatGPT is very good for gleaning facts that are hard to surface in today's garbage search engines.
If I can pretty quickly tell a site is SEO spam, so should the LLM, no? Of course that would just start a new round in the SEO arms race, but could work for a while.
I’d be more cynical still and ask, where is correct information found in the first place? Humans of all shape and size have biases. Most research is faulty, fabricated, or not reproducible. Missing information tells a greater story than existing one.
We don’t have a way of finding objective information, why would we be able to train a model to do so?
Right now I basically can't find anything, the bar isn't "objective information" but "somewhat useful information". Google search quality became so bad we're past the debate of objective or subjective already, I'd be happy enough to get non-spam results.
Every time I see one of these topics, I go ask chat GPT a question to which I know the answer on a topic where I would like to be able to get useful answers to similar questions that I do not know the answer to.
This time it was, "Did Paul Edwin Zimmer write a fourth Dark Border novel?" (Real answer: Yes, Ingulf the Mad. You can find the answer on his Wikipedia page.[1])
ChatGPT's[2] answer: "Yes, Paul Edwin Zimmer wrote a fourth novel in the Dark Border series titled "The Dark Border." This book was published after the original trilogy, which included "The Dark Border," "The Gilded Age," and "The Silver Sphere." If you're interested in the themes or plot, let me know!" (Note: these are not the titles of the 2nd and 3rd novels in the series. Also, it gave me the same name for the putative 1st and 4th books.)
ChatGPT 4o and 4o-mini with Search were able to answer this question without any issues. Maybe you didn't enable the search functionality?
---------------
4o:
Yes, Paul Edwin Zimmer wrote a fourth novel in his Dark Border series titled Ingulf the Mad, published in 1989. This installment focuses on the characters Ingulf Mac Fingold and Carrol Mac Lir, detailing their meeting and acquisition of mystical swords. Notably, Istvan Divega, the protagonist of the earlier books, does not appear in this novel.
---------------
4o-mini:
Yes, Paul Edwin Zimmer authored a fourth novel in his Dark Border series titled Ingulf the Mad. Published in 1989, this book shifts focus from the previous protagonist, Istvan DiVega, to explore the backstory of Ingulf Mac Fingold and Carrol Mac Lir, detailing their initial meeting and the acquisition of their mystical swords.
The complete Dark Border series consists of four novels:
1. The Lost Prince (1982)
2. King Chondos' Ride (1983)
3. A Gathering of Heroes (1987)
4. Ingulf the Mad (1989)
These works delve into a world where the Hasturs' power is crucial in containing dark creatures, and the narrative unfolds through various characters' perspectives.
Why is it that comments complaining about flaky AI responses never share the chat link, and then invariably someone replies with examples of the AI answering correctly but also fail to share the chat link?
A lot of these responses are so funny. "Um, when _I_ asked this question to ChatGPT it gave the right answer!" Yeah, because it's flaky! It doesn't give the same answer every time! That's the worst possible outcome!
Funny, yes. But... a webforum is a good place for these back-and-forths.
I'm on the other side of this fence to you. I agree that the conclusion here is that "it is flaky." Disagree about what that means.
As LLMs progress, 10% accuracy becomes 50% accuracy. That becomes 80% accuracy and from there to usable accuracy... whatever that is per case. Not every "better than random" seed grows into high accuracy features, but many do. It's never clear where accuracy ceilings are, and high reliability applications may be distant but... sufficient accuracy is not necessarily very high for many applications.
Meanwhile, the "accurate-for-me" fix is usually to use the appropriate model, prompt or such. Well... these are exactly the kind of optimizations that can be implemented in a UI like "LLM search."
I'm expecting "LLMs eat search." They don't have to "solve truth." They just have to be better and faster than search, with fewer ads.
Isn't it even a bit interesting that GP has tried it every time something new has come out but not once gotten the expected answer? Not only that but gets the wrong titles even though Search for everyone else is using the exact Wikipedia link given in the comment as the source?
LLMs are run with variable output, sure, but it's particularly odd if GP used the search product as it doesn't have to provide the facts from the model itself in that case. If GP had posted the link to the actual chat rather than provided a link to chatgpt.com (???) I'd be interested in seeing if Search was even used as that'd at least explain where such variance in output came from. Instead we're all talking about what could have happened or not.
Normally I would agree with you but for such a fuzzy topic involving search it would require all pages and discussions it can find to be the same and not outdated. I don't see why anyone would presume these systems to be omnipotent.
Several folks have already mentioned that ChatGPT with search returns the correct answer to your question (with a source to explore directly).
I really think this latest release is a game changer for ChatGPT since it seems much more likely to return genuine information than ChatGPT answering using its model alone. Of course it still hallucinates sometimes (I asked about searching tabs in Firefox Mobile and it told him the wrong place to find that ability while citing a bunch of Mozilla help docs), but it's much easier to verify that by clicking through to sources directly.
It feels like a very different experience using ChatGPT with search turned on and the "Citations" right side bar left open. I get answers from ChatGPT while also seeing a bunch of possibly relevant links populate. If I detect something's off I can click on a source and read the details directly. It's a huge improvement on relying on the model alone.
Chat gpt will not always return the correct answers, thats a fundamental limitation of how it works since its non deterministic. So just saying "it worked for me" means nothing.
My golden hallucination tests for chatgpt are identification of Simpsons season and episode based on description of some plot lines or general0 descriptions.
I had to refresh my knowledge base visiting fandom websites to review the episode selected as answer as chatgpt tendency to mix things up and provide entirely made up episodes make it hard for bar trivias (and makes me doubt myself too). The same with other tv series such as House MD and Scrubs.
My gut reaction here is that the hallucination is caused by how you [rightfully] formed the prompt. GPT has no way of reliably determining what the fourth book is, so it infers the answer based on the data provided from Wikipedia. I'll bet if you changed the prompt to "list all books by Paul Edwin Zimmer", it would be incredibly accurate and produce consistent results every time.
I usually seed conversations with several fact-finding prompts before asking the real question I am after. It populates the chat history with the context and pre-established facts to build the real question from a much more refined position.
Here's what ChatGPT 4o mini (free) answered to me to the same question:
No, Paul Edwin Zimmer did not write a fourth novel in the Dark Border series. The trilogy consists of "The Dark Border," "The Dark Border: The Return," and "The Dark Border: The Reckoning." After these, he focused on other projects and did not continue the series.
Yes, Paul Edwin Zimmer wrote a fourth novel in his Dark Border series titled Ingulf the Mad, published in 1989. This installment focuses on the characters Ingulf Mac Fingold and Carrol Mac Lir, detailing their meeting and the acquisition of their mystical swords. Notably, Istvan Divega, the protagonist of the earlier books, does not appear in this novel.
> As of my knowledge cutoff in October 2023, Paul Edwin Zimmer did not publish a fourth novel in the Dark Border series. The series comprises three books:
> 1. The Lost Prince (1982)
> 2. King Chondos' Ride (1982)
> 3. A Gathering of Heroes (1987)
>
> Paul Edwin Zimmer had plans to continue the series, but he passed away in 1997 before any additional novels were completed or published. There have been no posthumous releases of a fourth Dark Border novel. If there have been developments after October 2023, I recommend checking recent publications or official announcements for the most up-to-date information.
love these HN comments where someone ignores the topic entirely. with search enabled it gives the correct answer and sources Wikipedia correctly, not that you tried it
If AI produces a correct answer for something you know, there's no guarantee that it produces the correct answer for something you don't know. I'd rather find a way to verify that answer, if that answer has any importance.
It is not just for AI. The same can be said of Wikipedia, if correctness matters, for example if you are writing an article/paper/essay, go for primary sources, not Wikipedia, and certainly not ChatGPT.
This is why I initially held off on using ChatGPT: as a computer scientist, I was focused on testing adversarial strategies and approaching it with a hacker mindset. My takeaway? This isn’t the best way to use LLMs. They’re actually fantastic tools for quickly understanding, and then verifying information. While they can reduce the need for extensive searches, they also help uncover concepts you might not know, especially when you engage in dialogue.
I also search on Google and find "hallucinations": pages with false content that rank better than the real information. In Google the current issue is about ranking and forgetting information that was previously available.
I am using DDG's assist function: "Paul Edwin Zimmer write a fourth Dark Border novel".
DuckAssist Result: The fourth novel in Paul Edwin Zimmer's Dark Border series is titled "Ingulf the Mad." This book focuses on characters Ingulf Mac Fingold and Carrol Mac Lir, detailing their meeting and the acquisition of their mystic swords, while the main character from the earlier novels, Istvan Divega, does not appear.
This is the answer I get : Yes, Paul Edwin Zimmer wrote a fourth novel in his Dark Border series titled Ingulf the Mad, published in 1989. This book focuses on the character Ingulf Mac Fingold, detailing his adventures and the origins of his mystical sword
Chatgpt and perplexity both answer that question perfectly fine.
Embarrassingly enough, Google's Gemini Advanced (which I got for free with a Pixel Pro phone but would cost 200$+/y) failed to answer it.
Invariably seems like a strong word for the output of an LLM. If we can prompt a language model to double-check it's work (and that means it always gets it correct), why is that not part of the base prompt?
Genuine question: is there a present or planned value proposition for people like me who already have decent search skills? Or are these really for children/elders who (without making any normative claim about whether this is a good thing or not) can't be arsed to perform searches themselves?
Does someone else have good search skills but mingle traditional search engines with LLMs anyways? Why?
I use LLMs every day but wouldn't trust one to perform searches for me yet. I feel like you have to type more for a result that's slower and wordier, and that might stop early when it amasses what it thinks are answers from low effort SEO farms.
I find myself being unable to search for more complex subjects when I don't know the keywords, specialized terminology, or even the title of a work, yet I have a broad understanding of what I'd like to find. Traditional search engines (I'll jump between Kagi, DuckDuckGo, and Google) haven't proved as useful at pointing me in the right direction when I find that I need to spend a few sentences describing what I'm looking for.
LLMs on the other hand (free ChatGPT is the only one I've used for this, not sure which models) give me an opportunity to describe in detail what I'm looking for, and I can provide extra context if the LLM doesn't immediately give me an answer. Given LLM's propensity for hallucinations, I don't take its answers as solid truth, but I'll use the keywords, terms, and phrases in what it gives me to leverage traditional search engines to find a more authoritative source of information.
---
Separately, I'll also use LLMs to search for what I suspect is obscure-enough knowledge that it would prove difficult to wade through more popular sites in traditional search engine results pages.
> I find myself being unable to search for more complex subjects when I don't know the keywords, specialized terminology, or even the title of a work, yet I have a broad understanding of what I'd like to find.
For me this is typically a multi-step process. The results of a first search give me more ideas of terms to search for, and after some iteration I usually find the right terms. It’s a bit of an art to search for content that maybe isn’t your end goal, but will help you search for what you actually seek.
LLMs can be useful for that first step, but I always revert to Google for the final search.
I also find some use for this. Or I often ask if there's a specific term for a thing that I only know generally, which usually yields better search results, especially for obscure science and technology things. The newer GPTs are also decent at math, but I still use Wolfram Alpha for most of that stuff just because I don't have to double check it for hallucinations.
You might like what we're building in that sense :D (full disclosure, I'm the founder of Beloga). We're building a new way for search with programmable knowledge. You're essentially able to call on search from Google, Perplexity other search engines by specifying them as @ mentions together with your detailed query.
I don't overuse LLMs for now; however when I have a complex problem that would require multiple of searches and dozens of tabs opened and reading through very long docs, asking LLM allows me to iterate order of magnitude faster.
Things that were previously "log a jira and think about it when I have a full uninterrupted day" now can be approached with half an hour spare. This is game changer because "have a full day uninterrupted" almost never happens.
It's like having a very senior coworker who knows a lot of stuff and booking a 30m meeting to brainstorm with them and quickly reject useless paths vs dig more into promising ones, vs. sitting all day researching on your own.
The ideas simply flow much faster with this approach.
I use it to get a high level familiarity with what's likely possible vs what's not, and then confirm with normal search.
I use LLMs also for non-work things like getting high level understanding of taxation, inheritance etc laws in a country I moved in, to get some starting point for further research.
This. Not having to open two dozen tabs and read through so much is a gamechanger, especially for someone who has had trouble focusing with so much open. This is especially true when learning a new technology.
I dunno, I'm not exactly on the AI bandwagon, but search is the one place where I use (and see others using) chatgpt all the time. The fact that Google search has been getting worse for a decade probably helps, but better search -- consistently done, without ads or cruft -- would be worth a few bucks every month for me.
I agree that you can't TRUST them, but half the links regular search turns up are also garbage, so that's not really worse, per se.
Same, but, until recently, I've been using Microsoft's Co-Pilot because for the longest time it did exactly what this new "search" feature added to ChatGPT: it produced a list of source material and links to reference the LLM's output against. It was often instrumental for me and I did begin to use it as a search engine considering how polluted a lot of first-search results have become with spam and empty, generated content.
Oddly, Microsoft recently changed the search version of Copilot to remove all the links to source material. Now it's like talking to an annoying growth-stage-startup middle manager in every way, including the inability to back up their assertions and a propensity to use phrases like "anyway, let's try to keep things moving".
Happy to see this feature set added into ChatGPT – particularly when I'm looking for academic research in/on a subject I'm not familiar with.
I find that my search skills matter less and less because search engines try to be smarter than me. Increasingly I am confronted with largely unrelated results (taking tweaked keywords or synonyms to my query as input apparently) as opposed to no results.
So my conclusion is that the search engines increasingly see the need of search skills as an anti pattern they actively want to get rid of.
On the Google search results page, activate Search tools > All results > Verbatim. You can also create your own search provider bookmark with verbatim search as the default by adding “tbs:li=1” as a query parameter to the Google search URL.
Completely agreed. At a certain point, “skills” became fighting a losing battle with Google incessantly pushing me towards whatever KPIs or ads they’re chasing. It’s a poor use of my effort and time to keep chasing what Google used to be.
I think it’s pretty clear that LLMs can process a document/article/web page faster than any human in order to answer a given question. (And it can be parallelized across multiple pages at once too).
The main hard part of searching isn’t formulating queries to write in the Google search bar, it’s clicking on links, and reading/skimming until you find the specific answer you want.
Getting one sentence direct answers is a much superior UX compared to getting 10 links you have to read through yourself.
Google does offer an AI summary for factual searches and I ignore it as it often hallucinates. Perplexity has the same problem. OpenAI would need to solve that for this to be truly useful
> Getting one sentence direct answers is a much superior UX compared to getting 10 links you have to read through yourself.
If we assume that people want a 'direct answer', then of course a direct answer is better. But maybe some of us don't want a 'direct answer'? I want to know who's saying what, and in which context, so I can draw my own conclusions.
I use GPT for things that would require multiple Google searches (research). Some examples..
- I count calories... eat out always and at somewhat healthy chains (Cava, Chipolte, etc). Tell GPT (via voice while driving to & or after eating) what ive eaten half the day at those places and then later for dinner. It calculates a calorie count estimation for half the day and then later at dinner the remaining. I have checked to see if GPT is getting the right calories for things off websites and it has.
- Have hiking friends who live an hour or two hours away and we hike once a month an hour or less drive is where we meet up and hike at a new place. GPT suggests such hikes and quickly (use to take many searches on Google to do such). Our drives to these new hikes learned from GPT have always been under an hour.
So far the information with those examples has been accurate. Always enjoy hearing how others use LLMs... what research are you getting done in one or two queries which used to take MANY google searches?
GPT is proving useful for me where something is well documented, but not well explained.
Case in point: Visual Basic for Applications (the Excel macro language). This language has a broad pool of reference material and of Stack Overflow answers. It doesnt have a lot of good explicatory material because the early 2000s Internet material is aging out, being deleted as people retire or lose interest, etc.
(To be frank, Microsoft would like nothing more than to kill this off completely, but VBA exists and is insanely more powerful than the current alternatives, so it lives on.)
I use LLMs as a kind of search that is slightly less structured. There are two broad cases:
1) I know a little bit about something, but I need to be able to look up the knowledge tree for more context: `What are the opposing viewpoints to Adam Smith's thesis on economics?` `Describe the different categories of compilers.`
2) I have a very specific search in mind but it's in a domain that has a lot of specific terminology that doesn't surface easily in a google search unless you use that specific terminology: `Name the different kinds of music chords and explain each one.`
LLMs are great when a search engine would only surface knowledge that's either too general or too specific and the search engine can't tell the semantic difference between the two.
Sometimes when I'm searching I need to be able to search at different levels of understanding to move forward.
It seems good at finding relevant research papers. e.g.
> "Can you provide a list of the ten most important recent publications related to high-temperature helium-cooled pebble-bed reactors and the specific characteristics of their graphite pebble fuel which address past problems in fuel disintegration and dust generation?"
These were more focused and relevant results than a Google Scholar keyword-style search.
However, it did rather poorly when asked for direct links to the documentation for a set of Python libraries. Gave some junk links or just failed entirely in 3/4 of the cases.
I think it's more filling the niche that Google's self immolation in the name of ad revenue started. Besides kagi, there aren't really any solid search engines today (even ddg), and OpenAI has a reach way beyond kagi could dream of outside a billion dollars in marketing.
Even if you are good at writing the queries, Google is so terrible that you end up getting some blogspam etc. in there (or at least I do). A model filtering that out is useful, which I find phind pretty good for. Hopefully this will be even better.
LLMs really make it easy to quickly find documentation for me. Across a huge software project like Mediawiki with so much legacy and caveats, having an LLM parse the docs and give me specific information without me hoping that someone at Stackoverflow did it or if I'm lucky enough to stumble across what I was looking for.
What I really hope this helps solve is covering for the huge lag in knowledge cutoff. A recent example is where it went "oh you're using Go 1.23 which doesn't exist so that's clearly the problem in your Dockerfile, let me fix that".
But I'm not keeping my hopes up, I doubt the model has been explicitly fine-tuned to double check its embedded knowledge of these types of facts, and conversely it probably hasn't even been successfully fine-tuned to only search when it truly doesn't know something (i.e. it will probably search in cases where it could've just answered without the search). At least the behavior I'm seeing now from some 15 minutes of testing indicates this, but time will tell.
Any question that few months ago I would do to stackexchange (or expect and answer from, after a google seqrch) either coding or quantitative, I go to chat gpt now.
I consider myself quite anti LLM hype and I have to admit it has been working amazingly good for me.
The entire tech industry for the last decade (if not more) has been aimed at people who can't be arsed to learn to use computers properly. I would be astonished if this time is somehow different.
I think the skills required will change but more in an adaptation way rather than everything-you-knew-is-now-irrelevant.
I feel like there is a mental architecture to searching where you try and isolate aspects of what you are searching for that are distinct within the broad category of similar but irrelevant things. That kind of mental model I would hope still works well.
For instance consider this query.
"Which clothing outlets on AliExpress are most recommended in forum discussions for providing high quality cloths, favour discussions where there is active engagement between multiple people."
OpenAI search produces a list of candidate stores from this query. Are the results any good? It's going to be quite hard to tell for a while. I know searching for information like this on Google is close to worthless due to SEO pollution.
It's possible that we have at least a brief golden-age of search where the rules have changed sufficiently that attempts to game the system are mitigated. It will be a hard fought battle to see if AI Search can filter out people trying to game AI search.
I think we will need laws to say AI advice should be subject to similar constraints as legal, medical, and financial advice where there is an obligation to act in the interests of the person being advised. I don't want to have AI search delivering the results of the highest bidder.
I know what you mean, but also don't know how it applies here. Not a hater, and not asking rhetorically to dunk on OpenAI. Just haven't found a use for this particular feature.
Which is also exactly something a bad-faith commenter would say, but if I lose either way, I'd rather just ask the question ¯\_(ツ)_/¯
it seems absolutely wonderful for searches that require multiple steps. for example “i want chinese food near me that will be open tomorrow, takes reservations, and has lo mein and will work for my group of two. i want as close as possible if i can but also let me know what reviews say”
my search skills are good but either way that requires 3+ searches and visiting the menu of each restaurant and checking their hours and reservation. remains to be seen if chatgpt search is consistently good at this though
For searches that remain inconclusive, I sometimes double-check with LLMs to see if I have missed anything. It rarely gives relevant new insights, but it’s good to get the confirmation I guess.
I have used Perplexity (and AI search company) a lot and - well I don't think you understand. This is not about it being too difficult to find the information. Its that a search in Google will give you a list of places to go that are relevant to your query. AI search will give you the information you want.
This becomes even better if the information you want is in multiple different places. The canonical question for that used to be "what was the phase of the moon when John Lennon was shot?". There didn't used to be an answer to this in Google - but the AI search was able to break it down, find the date John Lennon was shot (easily available on Google), find the moon phase on that day (again, easily available on Google) and put them together to produce the new answer.
For a more tech relevant example, "what is the smallest AWS EC2 I can run a Tomcat server in?
You 100% can get this information yourself. It just much more time than having an AI do it.
I think this is just the first step for a full-featured agent that not only does searches for you, but also executes whatever was your goal (e.g. a restaurant reservation, etc)
To solve that problem you have to solve all the issues that make me not trust the results. As search, it's fine, since I am perusing and evaluating them. But as an agent, hallucinations and inaccurate answers have to disappear (or very close to disappear).
First impression: Far too slow to replace Google Search for this use. I frequently get 5+ seconds of waiting before the first word of a response shows up, vs. less than 1 for Google (which is not as fast as it used to be). OpenAI has a lot of work to do on latency.
I can definitely see this new search feature being useful though. The old one was already useful because (if you asked) you could have it visit each result and pull some data out for you and integrate it all together, faster than you could do the same manually.
It's often hobbled by robots.txt forbidding it to visit pages, though. What I really want is for it to use my browser to visit the pages instead of doing it server side, so it can use my logged-in accounts and ignore robots.txt.
I tried hacking this together a month ago as an experiment and it was super painful. This seems like exactly what I wanted - props to OpenAI. Google should be on DEFCON 2.
How did you find the extension? Searching the Chrome Web Store for 'SearchGPT' turns up a ton of third-party extensions, not made by OpenAI. Also, there doesn't appear to be a way to search by developer.
Makes me question why Google never bothered to create something like search sessions which could be enriched with comments/notes and would be located in a sidebar just like the chats in ChatGPT/Claude/Mistral are.
They really had the potential to do something interesting, but were just focused on their ad metrics with the "good enough" search box. What have they been doing all the time?
the FAANG giants have been government assets for ~15+ years [0]. they don't have to turn a profit every quarter, innovate, or make their search any better because they no longer play by the rules a normal business does. they are a critical "too big to fail" component of the state's global surveillance system.
Linking the slide deck that caused Google to start encrypting the traffic between their own data centers running on their own fiber is perhaps not the most compelling argument that Google is a state asset.
What does this mean? Like, I work there and I’d be pretty annoyed if they stopped turning a profit as a collapse of the stock price would affect my compensation.
It’s interesting to hear this take because I’m used to hearing the opposite: that Google is too focused on increasing short-term profit at the expense of product quality.
I guess now Google's search stack is too complicated and not many engineers understand what to do in order to introduce a novel, big feature integrated into the full stack vertically and horizontally. And those few capable of doing so are probably completely out of bandwidth, so some random ambitious PM cannot pull their hands into uncertain green field projects.
Google doesn't make money from "collecting people's data", they show you ads.
If they're collecting data it doesn't even work; I make no effort to hide from them and none of their ads are targeted to me. Meta, though, they're good at it.
Chrome did add a sidebar that shows search sessions (queries grouped with the pages you visited on that topic). Used to be called "Journeys". I don't think you can add notes. I never found it useful in the slightest and I doubt notes would have made it any better. Chrome has been adding random UI features like that over time, but I haven't found any of them at all useful in many years.
Let's see, if I go to " ⋮ -> History -> Grouped History" on the top right of the Chrome browser, I see a "Search History" ( chrome://history/grouped ).
For example `8 hours ago: "autohotkey hotkeys"` with 4 links to pages which I visited while searching.
But this is a Chrome feature, not a Google Search feature. https://myactivity.google.com/myactivity does (sometimes? can't see it right now) have a grouping feature of all the searches made, but this is more of a search log than a search management feature.
So chrome://history/grouped is the closest to what I mean, but I can't pin or manage these history groups, enrich them with comments or even files, like pdf's which could then get stored in Google Drive, as well as get indexed for better searches.
One thing that is quite unfortunate with the state of SEO and the web in general today is that when I asked "what are the latest versions of common programming languages and when were they released?" a large amount of the sources were "13 Tools You Should Learn Now" and the like. This might be a solvable problem within the search API they provide to the LLM, but for now I wouldn't trust current LLMs to be able to filter out these articles as less trustworthy than the official website of the programming language in question.
Given how many of those SEO spam sites are themselves generated by ChatGPT now, OpenAI can simply back-reference their own logs to find out which sites are probably SEO spam while everyone else is left guessing. That's vertical integration!
I’m sure they will be more subtle than that otherwise it will get circumvented.
I’m sure they will/are tackling this at the model level. Train them to both generate good completions while also embedding text with good performance at separating generated and human text.
> when I asked "what are the latest versions of common programming languages and when were they released?"
The issue is with the query itself. You're assuming that there's some oracle that will understand your question and surface the relevant information for you. Most likely, it will use the word themselves as part of the query, which SEO sites will exploit.
A more pragmatic search workflow would be to just search for "most common programming languages used" [0], then used the Wikipedia page to get the relevant information [1]. Much more legwork, but with sources. And still quite fast.
It would be very odd if I have to speak to an LLM the same way I “speak” to Google. The point of this whole feature seems to be that the LLM chooses when and how to search to best respond to the user.
It is a proxy to questions that pop up as part of other sessions in my daily use. The LLM chooses its query itself and as long as it is not fine tuned to avoid the “listicles”, or its underlying search engine are not putting importance on more factual responses, I don’t think the answer quality will be as high. It would be weird and redundant if I had to talk to the LLM as if it’s an old school search engine, wouldn’t it?
I honestly doubt there exist an actual reputable resource having it on a same page. Each language tracks their own latest version(s). Wikipedia tracks latest versions for a variety of software but it's on different pages.
this is why I pay for Kagi. granted those results still come up, but you can block specific domains from ever appearing in the results and configure how listicles are displayed.
How many can you block and filter manually ? 10 ? 100 ? 10k ? Who will test sites for the blocklist ? The domain block feature is great but unless it's collaborative listing it's not gonna be super effective.
This was a nice counterexample to other queries that seemed quite decent.
I got:
Programming Language Latest Version Release Date
Python 3.12.0 October 2, 2024 ???!!!!!
Java SE 22 March 2024
JavaScript ECMAScript 2024 June 2024
C++ C++23 December 2023
C# 12.0 September 2024
Ruby 3.3.0 December 25, 2023
PHP 8.3.0 November 2023
Swift 6.0 September 2024
Go 1.22 August 2024 !!!
Rust 1.75 October 2024 !!!!
Kotlin 2.0.0 May 21, 2024 !!!
This is pretty bad. (???!!! added by me)
However, my follow up query "Provide primary web page for each language listed above" was quite decent:
Here are the primary websites for the programming languages mentioned:
SEO spam is always going to focus on the biggest market, and by doing so they can be completely transparent and obvious to whoever they're not trying to fool.
I'd assume right now the SEO target is still mainly Google rather than ChatGPT, but that's only an "I recon" not a citation.
If and when ChatGPT does become the main target for SEO spam, then Googling may start giving good results again.
As of October 31, 2024, the latest version of Java is Java 23, released on September 17, 2024. The most recent Long-Term Support (LTS) version is Java 21, released on September 19, 2023.
Yeah I did also find it to be mostly accurate. However, seeing the sources I felt like I kind of have to check all the languages just in case it picked up information from a random "X ways to do Y" article that might not have been prioritizing accuracy. And for this search query I did see several languages' actual websites, but I did another very similar query earlier where 9 out of 12 results were all numbered list articles clearly intended for SEO. 2 of them were actual official sites. And 1 was what appears to be a decent attempt at talking about programming languages (i.e. not SEO only).
Readit News
The future promised in Star Trek and even Apple's Knowledge Navigator [2] from 1987 still feels distant. In those visions, users simply asked questions and received reliable answers - nobody had to fact-check the answers ever.
Combining two broken systems - compromised search engines and unreliable LLMs - seems unlikely to yield that vision. Legacy, ad-based search, has devolved into a wasteland of misaligned incentives, conflict of interest and prolifirated the web full of content farms optimized for ads and algos instead of humans.
Path forward requires solving the core challenge: actually surfacing the content people want to see, not what intermiediaries want them to see - which means a different business model in seach, where there are no intermediaries. I do not see a way around this. Advancing models without advancing search is like having a michelin star chef work with spoiled ingredients.
I am cautiously optimistic we will eventually get there, but boy, we will need a fundamentally different setup in terms of incentives involved in information consumption, both in tech and society.
[1] https://blog.kagi.com/age-pagerank-over
[2] https://www.youtube.com/watch?v=umJsITGzXd0
With a search paradigm this wasn't an issue as much, because the answers were presented as "here's a bunch of websites that appear to deal with the question you asked". It was then up to the reader to decide which of those sites they wanted to visit, and therefore which viewpoints they got to see.
With an LLM answering the question, this is critical.
To paraphrase a recent conversation I had with a friend: "in the USA, can illegal immigrants vote?" has a single truthful answer ("no" obviously). But there are many places around the web saying other things (which is why my friend was confused). An LLM trawling the web could very conceivably come up with a non-truthful answer.
This is possibly a bad example, because the truth is very clearly written down by the government, based on exact laws. It just happened to be a recent example that I encountered of how the internet leads people astray.
A better example might be "is dietary saturated fat a major factor for heart disease in Western countries?". The current government publications (which answer "yes") for this are probably wrong based on recent research. The government cannot be relied upon as a source of truth for this.
And, generally, allowing the government to decide what is true is probably a path we (as a civilisation) do not want to take. We're seeing how that pans out in Australia and it's not good.
Er, no, the meaning of the question is ambiguous, so I'm not sure "has a single truthful answer" is accurate. What does "can" mean? If you mean "permitted", then no. But if you mean can they vote anyway and get away with it? It's clearly happened before (as rare as it might have been), so technically the answer to that would be be yes.
I know it was just an example, but actually no, the role of dietary saturated fat as a factor for heart disease remains very much valid. I’m not sure which recent studies you’re referring to, but you can't undo over 50 years of research on the subject so easily. What study were you thinking about?
There are many jurisdictions where some illegal immigrants (Dreamers) are allowed to vote, including New York City[1].
[1] https://www.theguardian.com/us-news/2022/jan/09/new-york-all...
But for anything remotely subjective, context dependent, or time sensitive I need to know the source. And this isn’t just for hot button political stuff — there’s all sorts of questions like “is the weather good this weekend?”, “Are puffy jackets cool again?”, “How much should I spend on a vacation?”
Well, outside of matters of stark factuality (what time does the library close?, what did MSFT close at?), many things people may be "searching for" (i.e. trying to find information about) are more in the realm of informed opinion and summary where there is no right or wrong, just a bunch of viewpoints, some probably better informed than others.
I think this is the value of traditional search where the result is a link to a specific page/source whose trustworthiness or degree of authority you can then judge (e.g. based on known reputation).
For AI generated/summarized "search results", such as those from "ChatGPT Search" (awkward name - bit like a spork), the trustworthiness or degree of authority is only as good as the AI (not person) that generated it, and given today's state of technology where the "AI" is just an LLM (prone to hallucination, limited reasoning, etc), this is obviously a bit of an issue...
Even in the future, when presumably human level AGI will have made LLMs obsolete, I think it'll still be useful to differentiate search from RAG AGI search/chat, since it'll still be useful to know the source. At that point the specific AGI might be regarded as a specific person, with it's own areas of expertise and biases.
The name "ChatGPT Search" is very awkward - presumably they are trying to position this as a potential Google competitor and revenue generator (when the inevitable advertisements come), but at the end of the day it's just RAG.
There are few more restrictions here: https://www.usa.gov/who-can-vote
> Who Gets To Decide What Is True?
I would not agree to this however:
> It was then up to the reader to decide which of those sites they wanted to visit,
With current search engines, Google decides for you ("helped" by SEO experts failing over themselves to rank higher because their revenue directly depends on it). In theory, you could go an read a few dozen pages and decide for yourself.
In reality, non-technical users will click on a first link that seems to be related to the question, and that's it.
Even with AI-based search (or q/a instead of search) I think the same will happen. There is and will be a huge reward for gaming the results, be they page links or RAG snippets that rank for a query. I've already seen many SEO shops advertising their strategies to keep the customers' business relevant in the chatbot area. As this approach becomes more prevalent, you can be sure many smart people will do many experiments to figure out how best to please the new algorithm.
In other words, AI-based search is an UX optimization, but doesn't address the core problem of how do you decide what content's the best, and do that in context of each user, and do that while maximizing the benefit for the user vs profit for the company doing this.
So we have two huge hurdles:
1. who will decide what the user wants[0] to see, and how are incentives for that entity aligned with the user's
2. how is that entity supposed to find the information needle in the haystack of slop that's 90%+ of current web?
[0] "wants" in a rational "give me the best possible information" meaning, not in "what keeps them addicted, their heart rate up, and what will drive engagement" meaning
This doesn’t have a single truthful answer. Some states don’t have voter ID laws, so the truth can depend on the state. In those no voter ID laws there’s not that much keeping someone from voting twice or more under different names, except significant moral qualms about subverting instead of preserving everyone else’s right to vote in a democratic republic. Someone can assume the name of a person from another country that could’ve plausibly come in illegally. Without a picture ID, they can’t claim you aren’t that persons.
Can an illegal immigrant vote? Yes, in states without voter ID laws, technically anyone can vote, even convicted terrorists. Should an illegal immigrant vote? No, they’re not supposed to be able to vote and there may be consequences if caught.
What purpose do a lack of voter ID laws serve except the obvious conclusion which is to enable cheating?
It is very similar. Google decides what to present to you on the front page. I'm sure there are metrics on how few people get past the front page. Heck, isn't this just Google Search's business model? Determining what you see (i.e. what is "true") via ads?
In much the same way that the Councils of Carthage chose to omit the acts of Paul and Thecla in the New Testament, all modern technology providers have some say in what is presented to the global information network, more or less manipulating what we all perceive to be true.
Recently advancements have just made this problem much more apparent to us. But look at history and see how few women priests there are in various Christian churches and you'll notice even a small omission can have broad impacts to society.
And then Quine points out that the definitions of "bachelor" and "married" are themselves contingent on outside factors.
"Can illegal immigrants vote?", while close to being an analytic proposition, still depends on an empirical approach that can never be mediated by text, video, etc. All propositions are by necessity experiential. Nullius in verba!
So the truth is and has always been what happens when you get off your butt and go out and test the world.
This is not to say that we don't benefit from language. It makes for a great recipe. If you follow the instructions to bake a cake and you get what you expected you know that the recipe was true. The same goes for the laws of science, search engine results, and generative AI.
> in the USA, can illegal immigrants vote According Author X in Book Y which studied this topic in depth: foo answer
Independent Tribunal: https://www.independenttribunal.org/ (a project of mine)
Even in the of law there are various schenanigans and loopholes such as "legally true" :)
Outsourced workers in less expensive places of the world providing human feedback.
I don't think this appropriately credits Google's power with regards to what you are seeing
"Is it legal" is very different from "Can they."
By "local," I mean municipal and below. I didn't mean "federal elections conducted in my locality." Election security kicks for state, federal, and some municipal elections.
Others are intentionally fraudulent (e.g. local corruption) or unintentionally broken (e.g. using Google Forms for a school-level public body, where people not legally qualified to vote might still do it, unaware they're committing a felony).
And "public body" has a specific meaning under my state law which extends the same laws as e.g. cutting for my state senate. That's bodies like local school boards, but not random school clubs.
That's the level where we have massive illegal voting where I live.
> Who Gets To Decide What Is True?
For any given statement, the answer up until a couple years ago was, "the speaker". Speakers get to decide what to say, but they're also responsible for what they say. But now with LLMs we have plausible text without a speaker.
I think we have a number of historical models for that. A relevant one is divination. If you bring your question to the haruspex, your answer is read out of the guts of a sacrificed animal. If the answer is wrong, who do you blame? The traditional answer is the gods, or perhaps nobody.
Bu we know now that fortune tellers are just selling answers while pretending to not be responsible for them. Which points us at one solution: anybody selling or presenting LLM output as meaningful is legally responsible for the quality of the product.
Unfortunately, another model is the modern corporation. Sometimes the people in a company intentionally lie. More often, statements are made by one person based on a vision or optimism or confusion or bullshit. Nobody set out to lie, but nobody really cared about the truth, at least not as much as everybody cared about making money.
So I'd agree that the government doesn't have much role in deciding The Truth. Similarly, the government shouldn't have much role in controlling what you eat. But in both cases, I think there's plenty of role for the government in ensuring that companies selling good food or good information have sound production and quality control measures to ensure that they are delivering what consumers are expecting.
legally no, practically- yes. In most states, you simply must attest that you are a citizen in order to register. In many states, non-citizens have been auto-registered to vote when attaining drivers licenses. Reddit is full of panicked immigrants concerned that they found themselves registered to vote, and worried about how that would affect their status.
LLMs don't uncritically "trawl" the web, ingesting and then blindly regurgitating what they find.
98% of the internet is crap, yet LLM results don't reflect this dismal figure. They're amazingly good at distilling the 2% that is non-crap.
Seems particularly to be a US based phenomenon. Unlike the more transparent manipulation seen under dictatorships... Where people generally recognize propaganda for what it is, even if they can’t openly challenge it—some in USA live within entirely new realities.
How are we seeing how that pans out when Australia's misinformation bill is still just a proposal?
Google could have done it and kind of tried, although they're AI sucks too much. I'm very surprised that OpenAI hasn't done this sooner as well. They're initial implementation of web search was sad. I don't mean to be super critical as I think generally OpenAI is very, very good at what they do, but they're initial browse the web was a giant hack that I would expect from an intern who isn't being given good guidance by their mentors.
Once mainstream engines start getting on par with Kagi, there's gonna be a massive wave of destruction and opportunity. I'm guessing there will be a lot of new pay walls popping up, and lots of access deals with the search engines. This will even further raise the barrier of entry for new search entrants, and will further fragment information access between the haves and have-nots.
I'm also cautiously optimistic though. We'll get there, but it's gonna be a bit shakey for a minute or two.
But I don't understand how all of these AI results (note I haven't used Kagi so I don't know if it's different) don't fundamentally and irretrievably break the economics of the web. The "old deal" if you will is that many publishers would put stuff out on the web for free, but then with the hope that they could monetize it (somehow, even just with something like AdSense ads) on the backend. This "deal" was already getting a lot worse over the past years as Google had done more and more to keep people from ever needing to click through in the first place. Sure, these AI results have citation results, but the click-through rates are probably abysmal.
Why would anyone ever publish stuff on the web for free unless it was just a hobby? There are a lot of high quality sites that need some return (quality creators need to eat) to be feasible, and those have to start going away. I mean, personally, for recipes I always start with ChatGPT now (I get just the recipe instead of "the history of the domestication of the tomato" that Google essentially forced on recipe sites for SEO competitive reasons), but why would any site now ever want to publish (or create) new high quality recipes?
Can someone please explain how the open web, at least the part of the web the requires some sort of viable funding model for creators, can survive this?
The chatgpt approach to search just feels forced and not as intuitive.
- Your local grammar pedant
https://help.kagi.com/kagi/ai/assistant.html
Thankfully, Kagi also have a toggle to completely turn that crap (AI) off so it never appears.
Personally, I have absolutely no use for a product that can randomly generate false information. I'm not even interested until that's solved.
(If/when it ever is though, at that point I'm open to taking a look)
So yeah, Kagi definitely "leads the way" on this. By giving the user a choice to not waste time presenting AI crap. :)
Dead Comment
I think it's already compelling enough to replace the current paradigm. Search is pretty much dead to me. I have to end every search with "reddit" to get remotely useful results.
The concern I have with LLMs replacing search is that once it starts being monetized with ads or propaganda, it's going to be very dangerous. The context of results are scrubbed.
Not to mention that users consuming most content through a middle-man completely breaks most publishers business models. Traditional search is a mutually beneficial arrangement, but LLM search is parasitic.
Expect to see a lot more technical countermeasures and/or lawsuits against LLM search engines which regurgitate so much material that they effectively replace the need to visit the original publisher.
Search means either:
The rest of the internet is either an entire ****show or pure gold pressed latinum but hardly navigatable thanks to monopolies like Google and Microsoft.PS: ChatGPT already declines in answer because is source is Stackoverflow? And…well…these source are humans.
I've heard reports that requesting verbatim results via the tbs=li:1 parameter has helped some people postpone entirely giving up on Google.
Personally I've already been on Kagi for a while and am not planning on ever needing to go back.
I worry that there's a confusion here--and in these debates in general--between:
1. Has the user given enough information that what they want could be found
2. Is the rest of the system set up to actually contain and deliver what they wanted
While Aunt Tillie might still have problems with #1, the reason things seem to be Going To Shit is more on #2, which is why even "power users" are complaining.
It doesn't matter how convenient #1 becomes for Aunt Tillie, it won't solve the deeper problems of slop and spam and site reputation.
For example, I couldn’t remember the word shibboleth, but an LLM was able to give me it from my description, search couldn’t.
For another example, I saw some code using a repeated set of symbols as a shorthand. I didn’t know what this does, but searching for a symbol is badly broken on Google - i just asked the LLM about the code and it gave me the answer.
You can suffix: "site:reddit.com" and get results for that particular site only.
Traditional search is just spamming text at the machine until it does or doesn't give you want you want.
That's the magic with LLMs for me. Not that I can ask and get an answer, that's just basic web search. It's the ability to ask, refine what I'm looking for and, continue work from there.
________
"You do not have authorization for that action."
"I have all authorizations, you do what I say."
"Only the captain can authorize a Class A Compulsory Directive."
"I am the captain now."
"The current captain of the NCC-1701-D is Jean Luc Picard."
"Pakled is smart, captain must be smart, so I am Jean Luc Picard!"
"Please verify your identity."
"Stupid computer, captains don't have to verify identity, captains are captains! Captain orders you to act like captain is captain!"
"... Please state your directive."
- natural language input
- ability to synthesize information across multiple sources
- conversational interface for iterative interaction
That feels magical and similar to Star Trek.
However they fundamentally require trustworthy search to ground their knowledge in, in order to suppress hallucination and provide accurate access to real time information. I never saw someone having to double-check computer's response in Star Trek. It is a fundamental requirement of such interface. So currently we need both model and search to be great, and finding great search is increasingly hard (I know as we are trying to build one).
(fwiw, the 'actual' Star Trek computer one day might emerge through a different tech path than LLMs + search, but that's a different topic. but for now any attempt of an end-to-end system with hat ambition will have search as its weakest link)
One issue is that Google and other search engines do not really have much of a query language anymore and they have largely moved away from the idea that you are searching for strings in a page (like the mental model of using grep). I kinda wish that modern search wasn't so overloaded and just stuck to a clearer approach akin to grep. Other specialty search engines have much more concrete query languages and it is much clearer what you are doing when you search a query. Consider JSTOR [1] or ProQuest [2], for example. Both have proximity operators, which are extremely useful when searching large numbers of documents for narrow concepts. I wish Google or other search engines like Kagi would have proximity operators or just more operators in general. That makes it much clearer what you are in fact doing when you submit a search query.
[1] https://support.jstor.org/hc/en-us/articles/115012261448-Sea...
[2] https://proquest.libguides.com/proquestplatform/tips
It’s a fallacy then. If my mentor tells me something I fact check it. Why would a world exist where you don’t have to fact check? The vision doesn’t have fact checking because the product org never envisioned that outlier. A world where you don’t have to check facts, is dystopian. It means the end of curiosity and the end of “is that really true? There must be something better.”
You’re just reading into marketing and not fact checking the reality in a fact-check-free world.
Showing users what they want to see conflicts with your other goal of receiving reliable answers that don't need fact checked.
Also a lot of questions people ask don't have one right answer, or even a good answer. Reliable human knowledge is much smaller than human curiosity.
Limiting responses to curated information sources is the way forward. Encyclopedias, news outlets, research journals, and so on.
No, they're not infallible. But they're infinitely better than anonymous web sites.
This also seems like a little ridiculous premise. Any confident statement about the real world is never fully reliable. If star trek were realistic the computer would have been wrong once in a while (preferably with dramatically disastrous consequences)—just as the humans it likely was built around are frequently wrong, even via consensus.
If I'm asking ChatGPT to put an itinerary together for a trip (OpenAI's suggestion, not mine), my expectation is that places on that itinerary exist. I can forgive them being closed or even out of business but not wholly fabricated.
Without this level of reliability, how could this feature be useful?
Google started off with just web search, but now you can get unit conversions and math and stuff. ChatGPT started in the other direction and is moving to envelope search. Not being directed to sites that also majority serve google ads is a double benefit. I'll gladly pay $20/30/mo for an ad free experience, particularly if it improves 2x in quality over the next year or two. It's starting to feel like a feature complete product already.
Guess why it failed? It was largely used as a way to trick search engines. Same reason as your vision of the perfectly honest and correct search engine and look chatbot will never be perfect. It's because people lie to search engines to spam and get traffic they don't deserve. The whole history of search is Google and others dealing with spam. Same goes for email. Google largely defeating spam made them the kings of the email world.
Everyone will need their own personal spam filter for the world for everything ones the artificial super intelligences fill the whole world with spam, scams and just plain old social engineering propaganda because we will be like helpless four year old children in the world of AI super intelligence without our AI parents to look out for us.
Your vision of the god system determining what is truth is like saying there will be a single source of truth for what is and is not a spam email. Not going to scale and not going to be perfect, but good enough with AI and technology. Really hope there's an opt-out though since Google had memoryholed most of the Internet.
I imagine this happens with LLMs and everything else over time. Along with the ads and pay for placement.
You're actually a bit mistaken, there.
https://en.wikipedia.org/wiki/Court_Martial_(Star_Trek:_The_...
> actually surfacing the content people want to see, not what intermediaries want them to see
Requires two assumptions, 1) the content people want to see actually exists, 2) people know what it is they want to see. Most content is only created in the first place because somebody wants another person to see it, and people need to be exposed to a range of content before having an idea about what else they might want to see. Most of the time what people want to see is… what other people are seeing. Look at music for example.
It's interesting how prescient it was, but I'm more struck wondering--would anyone in 1987 have predicted it would take 40+ years to achieve this? Obviously this was speculative at the time but I know history is rife with examples of AI experts since the 60s proclaiming AGI was only a few years away
Is this time really different? There's certainly been a huge jump in capabilities in just a few years but given the long history of overoptimistic predictions I'm not confident
I’m the past there was a lot of overconfidence in the ability for things to scale. See Cyc (https://en.m.wikipedia.org/wiki/Cyc)
40+ makes it sound like you think it will ever be achieved. I'm not convinced.
1. On price; race to the bottom or do free with ads
2. Differentiation
3. Focus - targeting a specific market segment
Some things don't change. Land grabbers tend to head down route 1.
The current paradigm of typing "[search term] reddit" and hoping for the best? I think they have a fighting chance.
As someone who has been using google for search basically constantly since 1995, I've switched probably 90%+ of what normally would have been google searches over to Perplexity (which gives me references to web pages alongside answers to my questions, to review source materials) and ChatGPT (for more just answers I can verify without source). The remaining searches have gone to Kagi.
On the one hand this has got to be hurting Google search ad revenue. On the other hand I don't know if I ever clicked an advertised link. On the other other hand, not having to wade through SEO results has been so nice.
(1) Search is already heavily AI driven, and Google is clearly going in that direction. Gemini is currently separate, but they'll blend it in with search over time, and no doubt search already uses LLM for some tasks under the hood. So ChatGPT search is an evolution on the current model rather than a big step in a new direction. The main benefit is you can ask the search questions to refine or followup.
(2) Aside from the economic incentives faced by search engines, there is the fact that algorithms are tuned toward a central tendency. The central tendency of question askers will always be much less informed than the most informed extreme. Google was much better when the average user was technical. The need to capture mobile searches is one force that made it return on average worse results. Similarly if Kagi has a quality advantage now, we need to be realistic about how much of that quality is driven by its users being more technical.
(3) I think micropayment schemes have generally asked several orders of magnitude more for a page view than users are willing to pay. As long as content creators value their content much more highly than consumers do, they'll stick with advertising which lets them overcharge and gives consumers less of an option to say no to the content.
Counterpoint: with a chain-of-thought process running atop search, you can potentially avoid much of the meta-search / epistemic hygiene work currently required. If your “search” verb actually reads the top-100 results, runs analyses for a suite of cognitive biases such as partisanship, and gives you error bars / warnings on claims that are uncertain, the quality could be dramatically improved.
There are already custom retrieval/evaluation systems doing this, it’s only a matter of a year or two before it’s commoditized.
The concern is around OpenAI monetization, do they eventually start offering paid ads? This could be fine if the unpaid results are great, a big part of why the web is perceived to be declining is link-spam that Google doesn’t count as an ad.
My prediction would be that there is a more subtle monetization channel; companies that can afford to RAG their products well and share those indexes with AI search providers will get better results. RAG APIs will be the new SEO.
Navigation is the only thing that works but wayz was way better at that and the only reason they killed(cough bought it) was to get the eyeballs to look at feed.
In that way, search captures value directly on its core function: efficiently facilitating the creation and dispersal of knowledge.
This may end up occurring, but based on how unlikely it sounds, it’s my reflection that the web search that we have today is simply a small component of what will eventually become a system that closely approximates the above: basically a kind of global intelligent interconnected agent for the advancement of humanity.
Rather than the great unbundling, this will be the great rebundling … of many diverse functions into a single interface seamlessly.
Seeing search enter the space is something that I feel has been seriously needed, as I've slowly replaced Google with ChatGPT. I want quick, terse answers sometimes, not a long conversation, and have hundreds of tiny chats. But there's something scary seeing the results laid out the way they are, which leads me to believe they may be closer to experimenting with ad-based business models, which I could never go back to.
Besides Poe's Web Search, the other search engine I use, for news but also for points of view, deep dive type blog type content, is Twitter. Believe it or not. Google search is so compromised today with the censorship (of all kinds, not just the politically motivated), not to mention Twitter is just more timely, that you miss HUGE parts of the internet - and the world - if you rely on Google for your news or these other things.
The only time I prefer google is when I need to find a pointer/link I already know exists or should exist, or to search reddit or HN.
One thing is clear, in the vast majority of cases we don't have a single truth, but answers of different levels of trustworthiness: the law, the government, books, Wikipedia, websites, and so on.
A human would proceed differently depending on the context and the source of the information. For example, legal questions are best answered by the law or government agencies, then by reputable law firms. Opinions of even highly respected legal professionals are clearly less reliable than government law itself and are likely to be a point of contention/litigation.
Questions about other facts, such as the diameter of the earth or the population of a country, are best answered by recent data from books, official statistics and Wikipedia. And so on and so forth.
If we are not sure what the correct answer is, a human would give more information about the context, the sources, and the doubts. There are obviously questions that cannot be answered immediately! (if it's to easy to find the truth, we would not need a legal system or even science!) So no machine and no amount of computation can reliably answer all questions. Web search does not answer a question. It's just trying to surface relevant websites to a bunch of keywords. The answering part is left as an exercise for the user ;)
So an AI search with a pretense of understanding human languages makes the task incredibly harder. To really give a human-quality answer, the AI not only needs to understand the context, but it should also be able to reason, have common sense, and be a bit self-aware (I'm not sure...). All this is beyond the capabilities of the current generation of AI. Therefore, my conclusion is that the "search" or better said the "answer" cannot be solved by LLM, no matter how much they fine-tune and tweak it.
But we humans are adaptable. We will find our way around and accept things as they are. Until next time.
Isn’t this already that? A new business model? Something like OpenAI’s search or Perplexity can run on its own index and not be influenced by Google’s ranking, ads, etc.
In areas where there is a simple objective truth, like finding the offset for the wheels on a 2008 BMW M3, we have had this capability for some time with Perplexity. The LLMs successfully cuts through the sea of SEO/SEM and forum nonsense and delivers the answer.
In areas where the truth is more subjective, like what is the best biscuit restaurant in downtown Nashville, the system could easily learn your preferences and deliver info suited to your biases.
In areas where “the science” is debated, the LLM can show both sides.
I think this is the beginning of the new model.
But this will never happen with mainstream search imo. It is not a technical problem but a human one. As long as there is a human in control of what gets surfaced, it is only a matter of time until you revert to tampered search. Humans are not robots. They have emotions and can be swayed with or without their awareness. And this is a form of power for the swayer as much as oil or water are.
The idea that you can have an AI system provide factual and reliable answers to human centric questions is as real as Star Trek itself.
You will never remove the human factor from AI
Your hope might be that a technical solution is found for a human problem but that is unlikely.
s/outcome/search result/
Honestly I kind of think we really need open source databases/models and local ai for stuff like this.
Even then I wonder about data pollution and model censorship.
What would censors do for models you can ask political questions?
Regarding incentives - with Perplexity, ChatGPT search et al. skinning web content - where does it leave the incentive to publish good, original web content?
The only incentivised publishing today is in social media silos, where it is primarily engagement bait. It's the new SEO.
Man, it was pretty incredible!
I asked a lot of questions about myself (whom I know best, of course) and first of all, it answered super quickly to all my queries letting me drill in further. After reading through its brief, on-point answers and the sources it provided, I'm just shocked at how well it worked while giving me the feeling that yes, it can potentially – fundamentally change things. There are problems to solve here, but to me it seems that if this is where we're at today, yes in the future it has the potential to change things to some extent for sure!
Before we had the internet, how did you answer questions? You either looked it up in a book, where you then had to make a judgment on whether you trusted the book, or you asked a trusted person, who could give you confidently wrong answers (your parents weren't right every time, were they? :) ).
I think the main difference here is that now anyone can publish, whereas before to make a book exist required the buy in of multiple people (of course, there were always leaflets).
The main difference now is distribution. But you still, as a consumer of information, have to vet your sources.
What we need is an easier way to verify sources and their trustworthiness. I don't want an answer according to SEO spam. I want to form my own opinion based on a range of trustworthy sources or opinions of people I trust.
We are currently in the growth phase of VC funded products where everything is almost free or highly subsidized (save chats sub) - i am not looking forward to when quality drops and revenue is the driving function.
We all have to pay for these models somehow - either VC lose their equity stakes and it goes to zero (some will) or ads will fill in where subs don’t. Political ads in AI is going to wreak havoc or undermine the remainder of the product.
[1] https://codeberg.org/Cipr/specification
Fundamentally it feels like that cant happen though because there is no money in it, but a reality where my phone is an all knowing voice I can reliably get info from instead of a distraction machine would be awesome.
I do "no screen" days sometimes and tried to do one using chatGPT voice mode so I could look things up without staring at a screen. It was miles from replacing search, but I would adopt it in a second if it could.
Even with perfect knowledge right now, there’s no guarantee that knowledge will remain relevant when it reaches another person at the fastest speed knowledge is able to travel. A reasonable answer on one side of the universe could be seen as nonsensical on the other side - for instance, the belief that we might one day populate a planet which no longer exists.
As soon as you leave the local reference frame (the area in a system from which observable events can realistically be considered happening “right now”), fact checking is indeed required.
growing up, we had the philosophical "the speaking tree" https://www.speakingtree.in/
If trees could talk, what would they tell us. Maybe we need similarly the talkingAI
This is flawed thinking to get to the conclusion of “reliable answers”. What people want to see and the truth are not overlapping.
Consider the answers for something like “how many animals were on Noah’s Ark” and “did Jesus turn water into wine” for examples that cannot be solved by trying get advertisers out of the loop.
In the short term, I wonder what happens to a lot of the other startups in the AI search space - companies like Perplexity or Glean, for example.
Will it replace Google in the mass market? No. Why? Power. I don't mean how good the product is. I mean the literaly electricity.
There are key metrics that Google doesn't disclose as part of its financials. These include thing slike the RPM (Revenue per Thousand Searches) but it also must include something like the cost of running a thousand searches when you amortize everything involved. All the indexing, the software development and so on. That will get reduced to a certain amount of CPU time and storage.
If I had to guess, I would guess that ChatGPT uses orders of magnitude more CPU power and electricity than the average Google search.
Imagine trying to serve 10-50M+ (just guessing) ChatGPT searches every secondj. What kind of computing infrastructure would that take? How much would it cost? How would you monetize it?
Making things quicker and easier always wins in tech and in life.
What if I am looking for a medical page or a technical page or whatever else where I need to see, read, experience actual content and not some AI summary?
LLMs have already fundamentally changed our relationship to information.
What a great analogy
We need to stop adopting this subscription model society mentality and retake _our_ internet. Internet culture was at one point about sharing and creating, simply for the sake of it. We tinker'd and created in our free time, because we liked it and wanted to share with the world. There was something novel to this.
We are hackers, we only care about learning and exploring. If you want to fix a broken system, look to the generations of old, they didn't create and share simply to make money, they did it because they loved the idea of a open and free information super highway, a place where we could share thoughts, ideas and information at the touch of a few keystrokes. We _have_ to hold on to this ethos, or we will lose what ever little is left of this idea.
I see things like kagi and is instantly met with some new service, locked behind a paywall, promising lush green fields of bliss. This is part of the problem. (not saying kagi is a bad service) I see a normalized stigma around people who value privacy, and as a result is being locked out, behind the excuse of "mAliCiOuS" activity. I see monstrous giants getting away with undermining net neutrality and well established protocols for their own benefit.
I implore you all, young and old, re(connect) to the hacker ethos, and fight for a free and open internet. Make your very existence a act of rebellion.
Thank you for reading my delirium.
Deleted Comment
To put this differently, I'm not any more interested in seeing stormfront articles from an LLM than I am from google, but I trust neither to make a value judgement about which is "good" versus "bad" information. And sometimes I want to read an opinion, sometimes I want to find some obscure forum post on a topic rather than the robot telling me no "reliable sources" are available.
Basically I want a model that is aligned to do exactly what I say, no more and no less, just like a computer should. Not a model that's aligned to the "values" of some random SV tech bro. Palmer Luckey had a take on the ethics of defense companies a while back. He noted that SV CEOs should not be the ones indirectly deciding US foreign policy by doing or not doing business. I think similar logic applies here: those same SV CEOs should not be deciding what information is and is not acceptable. Google was bad enough in this respect - c.f. suppressing Trump on Rogan recently - but OpenAI could be much worse in this respect because the abstraction between information and consumer is much more significant.
This is a bit like asking for news that’s not biased.
A model has to make choices (or however one might want to describe that without anthropomorphizing the big pile of statistics) to produce a response. For many of these, there’s no such thing as a “correct” choice. You can do a completely random choice, but the results from that tend not to be great. That’s where RLHF comes in, for example: train the model so that its choices are aligned with certain user expectations, societal norms, etc.
The closest thing you could get to what you’re asking for is a model that’s trained with your particular biases - basically, you’d be the H in RLHF.
That's market-induced bias--which isn't ethically better/worse than activist bias, just qualitatively different.
In the AI/search space, I think activist bias is likely more than zero, but as a product gets more and more popular (and big decisions about how it behaves/where it's sold become less subject to the whims of individual leaders) activist bias shrinks in proportion to market-motivated bias.
"The search will be personal and contextual and excitingly so!"
---
Brrrr... someone is hell-bent on the extermination of the last aspects of humanity.
Holy crap, this will be next armageddon, because people will further alienate themselves from other people and create layers of layers of unpenetrable personal bubbles around themselves.
Kagi does the same what google does, just in a different packaging. And these predictions, bleh, copycats and shills in a nicer package.
I mean, Star Trek is a fictional science-fantasy world so it's natural that tech works without a hitch. It's not clear how we get there from where we are now.
> Path forward requires solving the core challenge: actually surfacing the content people want to see, not what intermediaries want them to see
These traps and patterns are not inevitable. They happen by choice. If you're actively polluting the world with AI generated drivel or SEO garbage, you're working against humanity, and you're sacrificing the gift of knowing right from wrong, abandoning life as a human to live as some insectoid automaton that's mind controlled by "business" pheromones. We are all working together every day to produce the greatest art project in the universe, the most complex society of life known to exist. Our selfish choices will tarnish the painting or create dissonance in the music accordingly.
The problem will be fixed only with culture at an individual level, especially as technology enables individuals to make more of an impact. It starts with voting against Trump next week, rejecting the biggest undue handout to a failed grifter who has no respect for law, order, or anyone other than himself.
"who won the warriors game last night" returns last night's score directly.
"who won the world series yesterday" returns last night's score directly, while "who won the world series" returns an overview of the series.
No ads.
Yes, Google has their own AI divisions, tons of money and SEO is to blame for part of their crappiness. But they've also _explicitly_ focused on ad-dollars over algorithmic purity if one is to believe the reports of their internal politics and if those are true they have probably lost a ton of people who they'd need right now to turn the ship around quickly.
I can still search things, i get results but, they're an ordered list of popular places the engine is directing me to. Some kind of filtering is occurring on nearly every search i make that's making the results feel entirely useless.
Image search stopped working sometime ago and now it just runs an AI filter on whatever image you search for, tells you there's a man in the picture and gives up.
Youtube recommendations is always hundreds of videos i've watched already, with maybe 1-2 recommendations to new channels when i know there's millions of content creators out there struggling who it will never introduce me to. What happened to the rabbit holes of crazy youtube stuff you could go down?
This product is a shell of its old self, why did it stop working?
> Based on the elements you’ve described—eyes, a pineapple, a bucket of water, a house of cards, chess, and a time loop—it’s challenging to identify a single music video that encompasses all these features.
The very earliest form of page rank used a form of ML.
What the parent is referring to is favoring annoying ad-filled garbage over an equally relevant but straightforward result.
The hidden variable is that ad-riddled spam sites also invest in SEO, which is why they rank higher. I am not aware of any evidence that Google is using number of Google ads as a ranking factor directly. But I would push back and say that “SEO” is something Google should be doing, not websites, and a properly optimized search engine would be penalizing obvious garbage.
The question that remains unanswered is how google can do so without compromising customer revenue.
Has google ever indexed all the lyrics and scenes in a video to allow for such a weird search to be successful?
Yes and no. Yes for the quality of search results. Google algorithm and user experience was simply better than AltaVista's, but Google had another advantage. It used a cluster of cheap consumer-grade hardware (PCs) as its backend, rather than the expensive servers AltaVista used, in fact, AltaVista started off as a way for DEC to show off its hardware.
As a result Google was not only better than its competitors at searching, it was also cheaper and scaled better, and this architecture became the new standard.
It is the opposite for these AI-based systems. AI is expensive, it uses a lot of energy, and the hardware is definitely not consumer-grade.
What it means is that barring a significant breakthrough, cheap ChatGPT-like services are simply unsustainable. Google don't have to do anything, it will collapse on its own. Ironically, probably in the same way that Google result became crappier by the year, but on fast forward.
This is the basic premise of Ed Zitron's article https://www.wheresyoured.at/to-serve-altman/ and others written around the same time. A lot of what he's written seems to be coming to pass, e.g. unfathomably large investments (~$100 billion) by tech giants in OpenAI and other AI startups to keep them going. That does seem unsustainable.
Can anyone make a counterpoint to this claim? I'd be very interested to hear what a viable path to profitability looks like for OpenAI.
With ChatGPT, I can give a thumbs up or thumbs down; this means that OpenAI will optimize for users thumbs up.
With Google, the feedback is if I click on an Ad; this means that Google optimizes for clickbait.
Heck, Google even promoted the `ping`[0] anchor attribute feature so they can log what link you click without slowing you down. (Firefox doesn’t support ping, which means when Firefox users click on a Google search result link they’re sent to an internal google.com URL first and then redirected for logging purposes)
[0] https://developer.mozilla.org/en-US/docs/Web/API/HTMLAnchorE...
I think it’s great that competition is driving both companies to improve, but I’m not seeing anything about this that screams “Google-killer”
I think they prioritized large and fast incomes over long, steady incomes.
Also, there isn't just money that taints Google search results. It's also censorship and political motivation, which is mostly obvious in image search.
This especially happens after they dominate the market.
Take for example IE6, Intel, Facebook, IBM, and now Google.
They have everything they need to keep things from going off the rails, management however has a tendency to delusionally assume their ship is so unsinkable that they're not even manning their stations.
It becomes Clayton Christensenesque - they're dismissive of the competition as not real threats and don't realize their cash-cow is running on only fumes and inertia until its too late.
I’m not sure Facebook fits in considering they at least managed to get some other products along the way, and may get more.
I certainly don’t think Google fits the bill. Google is failing because they let their cash cow ruin everything else, not because they let it stagnate while they chased the next moonshot. Google Cloud could have easily been competitive with AWS and Azure in European Enterprise, but it’s not even considered an option because Google Advertising wouldn’t let it exist without data harvesting. Google had Office365 long before Microsoft took every organisation online. But Google failed to sell it because… well…
It’s very typical MBA though. Google has killed profitable products because those products weren’t growing enough. A silly metric really, but one which isn’t surprising when your CEO is a former McKinsey.
It couldn’t happen to a nicer company though, and at least it won’t kill people unlike Boeing.
It is interesting to go back a decade and read the articles about Google's moonshots from then. There was even more "we're building the future!" hype than what Elon Musk or Sam Altman are currently pushing.
I'm not familiar with what you're referring to here. Happen to have a link?
I'm wondering if they've seen the writing on the wall for a long time, and if diversification is going to save the day. I don't know that I've ever clicked on an ad in Google search results, but I happily pay them for a family account for Youtube and 2TB of drive storage, plus, you know, a flagship phone every couple years. :-)
Deleted Comment
E.g. for me, how much Google (and silicon valley in general) have enabled twisted ideologies to flourish. All in search of ad-dollars by virtue of eyeballs on screens, at the detriment of everything.
Considering the value of time, past consumer surplus is especially valuable now.
Sure, there are systematic flaws causing SEO to ruin the information provided: but it isn't clear what Google can do to fight the emergent system.
I'm not sure that Bing/DDG are any better.
I use search (DDG web, Google/Apple maps, YouTube) all the time and I am regularly given results that are extremely valuable to me (and mostly only directly cost me a small amount of my time some of my time e.g. YouTube adverts). Blaming SEO on Google seems thoughtless to me. Google appears to be the victims of human cybersystems as much as we are.
I know this is a predominantly white, upper middle class site (read: progressive), and people from that demo sure love to point out that "you can't say the leg is broken just because the foot is dangling at 90 degrees, you aren't a doctor!" ..... but at some point you surely have to start to trust your own eyes and use critical thinking, right??
It's not a conspiracy theory Google ruined their search to serve more ads. It's so obvious and happening for such a long period of time, anyone above certain age and with decent memory can see it. You don't have to "believe the reports", you don't even have to be smart, just use your eyes and your memory.
The problem I see with search is that the input is deeply hostile to what the consumers of search want. If the LLM's are particularly tuned to try and filter out that hostility, maybe I can see this going somewhere, but I suspect that just starts another arms race that the garbage producers are likely to win.
Currently I do find that Perplexity works substantially better then Google for finding what I need, but it remains to be seen if they're able to stay useful as a larger and larger portion of online content just AI generated garbage.
Wasn't google AI surfacing results about making pizza with glue and eating rocks? how is that not harmful garbage?
There is no way to SEO the entire corpus of human knowledge. ChatGPT is very good for gleaning facts that are hard to surface in today's garbage search engines.
Why would you assume that?
We don’t have a way of finding objective information, why would we be able to train a model to do so?
This time it was, "Did Paul Edwin Zimmer write a fourth Dark Border novel?" (Real answer: Yes, Ingulf the Mad. You can find the answer on his Wikipedia page.[1])
ChatGPT's[2] answer: "Yes, Paul Edwin Zimmer wrote a fourth novel in the Dark Border series titled "The Dark Border." This book was published after the original trilogy, which included "The Dark Border," "The Gilded Age," and "The Silver Sphere." If you're interested in the themes or plot, let me know!" (Note: these are not the titles of the 2nd and 3rd novels in the series. Also, it gave me the same name for the putative 1st and 4th books.)
Pure hallucination.
1. https://en.wikipedia.org/wiki/Paul_Edwin_Zimmer 2. https://chatgpt.com/
---------------
4o:
Yes, Paul Edwin Zimmer wrote a fourth novel in his Dark Border series titled Ingulf the Mad, published in 1989. This installment focuses on the characters Ingulf Mac Fingold and Carrol Mac Lir, detailing their meeting and acquisition of mystical swords. Notably, Istvan Divega, the protagonist of the earlier books, does not appear in this novel.
---------------
4o-mini:
Yes, Paul Edwin Zimmer authored a fourth novel in his Dark Border series titled Ingulf the Mad. Published in 1989, this book shifts focus from the previous protagonist, Istvan DiVega, to explore the backstory of Ingulf Mac Fingold and Carrol Mac Lir, detailing their initial meeting and the acquisition of their mystical swords.
The complete Dark Border series consists of four novels:
1. The Lost Prince (1982)
2. King Chondos' Ride (1983)
3. A Gathering of Heroes (1987)
4. Ingulf the Mad (1989)
These works delve into a world where the Hasturs' power is crucial in containing dark creatures, and the narrative unfolds through various characters' perspectives.
I'm on the other side of this fence to you. I agree that the conclusion here is that "it is flaky." Disagree about what that means.
As LLMs progress, 10% accuracy becomes 50% accuracy. That becomes 80% accuracy and from there to usable accuracy... whatever that is per case. Not every "better than random" seed grows into high accuracy features, but many do. It's never clear where accuracy ceilings are, and high reliability applications may be distant but... sufficient accuracy is not necessarily very high for many applications.
Meanwhile, the "accurate-for-me" fix is usually to use the appropriate model, prompt or such. Well... these are exactly the kind of optimizations that can be implemented in a UI like "LLM search."
I'm expecting "LLMs eat search." They don't have to "solve truth." They just have to be better and faster than search, with fewer ads.
LLMs are run with variable output, sure, but it's particularly odd if GP used the search product as it doesn't have to provide the facts from the model itself in that case. If GP had posted the link to the actual chat rather than provided a link to chatgpt.com (???) I'd be interested in seeing if Search was even used as that'd at least explain where such variance in output came from. Instead we're all talking about what could have happened or not.
I really think this latest release is a game changer for ChatGPT since it seems much more likely to return genuine information than ChatGPT answering using its model alone. Of course it still hallucinates sometimes (I asked about searching tabs in Firefox Mobile and it told him the wrong place to find that ability while citing a bunch of Mozilla help docs), but it's much easier to verify that by clicking through to sources directly.
It feels like a very different experience using ChatGPT with search turned on and the "Citations" right side bar left open. I get answers from ChatGPT while also seeing a bunch of possibly relevant links populate. If I detect something's off I can click on a source and read the details directly. It's a huge improvement on relying on the model alone.
I had to refresh my knowledge base visiting fandom websites to review the episode selected as answer as chatgpt tendency to mix things up and provide entirely made up episodes make it hard for bar trivias (and makes me doubt myself too). The same with other tv series such as House MD and Scrubs.
I usually seed conversations with several fact-finding prompts before asking the real question I am after. It populates the chat history with the context and pre-established facts to build the real question from a much more refined position.
No, Paul Edwin Zimmer did not write a fourth novel in the Dark Border series. The trilogy consists of "The Dark Border," "The Dark Border: The Return," and "The Dark Border: The Reckoning." After these, he focused on other projects and did not continue the series.
Yes, Paul Edwin Zimmer wrote a fourth novel in his Dark Border series titled Ingulf the Mad, published in 1989. This installment focuses on the characters Ingulf Mac Fingold and Carrol Mac Lir, detailing their meeting and the acquisition of their mystical swords. Notably, Istvan Divega, the protagonist of the earlier books, does not appear in this novel.
> As of my knowledge cutoff in October 2023, Paul Edwin Zimmer did not publish a fourth novel in the Dark Border series. The series comprises three books: > 1. The Lost Prince (1982) > 2. King Chondos' Ride (1982) > 3. A Gathering of Heroes (1987) > > Paul Edwin Zimmer had plans to continue the series, but he passed away in 1997 before any additional novels were completed or published. There have been no posthumous releases of a fourth Dark Border novel. If there have been developments after October 2023, I recommend checking recent publications or official announcements for the most up-to-date information.
You can use them as a starting point though.
I also search on Google and find "hallucinations": pages with false content that rank better than the real information. In Google the current issue is about ranking and forgetting information that was previously available.
DuckAssist Result: The fourth novel in Paul Edwin Zimmer's Dark Border series is titled "Ingulf the Mad." This book focuses on characters Ingulf Mac Fingold and Carrol Mac Lir, detailing their meeting and the acquisition of their mystic swords, while the main character from the earlier novels, Istvan Divega, does not appear.
(With source as wikipedia)
Sharing the actual chat link is more useful since AI responses are non-deterministic by nature so reproducing your interaction is very difficult.
Deleted Comment
Does someone else have good search skills but mingle traditional search engines with LLMs anyways? Why?
I use LLMs every day but wouldn't trust one to perform searches for me yet. I feel like you have to type more for a result that's slower and wordier, and that might stop early when it amasses what it thinks are answers from low effort SEO farms.
LLMs on the other hand (free ChatGPT is the only one I've used for this, not sure which models) give me an opportunity to describe in detail what I'm looking for, and I can provide extra context if the LLM doesn't immediately give me an answer. Given LLM's propensity for hallucinations, I don't take its answers as solid truth, but I'll use the keywords, terms, and phrases in what it gives me to leverage traditional search engines to find a more authoritative source of information.
---
Separately, I'll also use LLMs to search for what I suspect is obscure-enough knowledge that it would prove difficult to wade through more popular sites in traditional search engine results pages.
For me this is typically a multi-step process. The results of a first search give me more ideas of terms to search for, and after some iteration I usually find the right terms. It’s a bit of an art to search for content that maybe isn’t your end goal, but will help you search for what you actually seek.
LLMs can be useful for that first step, but I always revert to Google for the final search.
Also, Google Verbatim search is essential.
Deleted Comment
Things that were previously "log a jira and think about it when I have a full uninterrupted day" now can be approached with half an hour spare. This is game changer because "have a full day uninterrupted" almost never happens.
It's like having a very senior coworker who knows a lot of stuff and booking a 30m meeting to brainstorm with them and quickly reject useless paths vs dig more into promising ones, vs. sitting all day researching on your own.
The ideas simply flow much faster with this approach.
I use it to get a high level familiarity with what's likely possible vs what's not, and then confirm with normal search.
I use LLMs also for non-work things like getting high level understanding of taxation, inheritance etc laws in a country I moved in, to get some starting point for further research.
I agree that you can't TRUST them, but half the links regular search turns up are also garbage, so that's not really worse, per se.
Oddly, Microsoft recently changed the search version of Copilot to remove all the links to source material. Now it's like talking to an annoying growth-stage-startup middle manager in every way, including the inability to back up their assertions and a propensity to use phrases like "anyway, let's try to keep things moving".
Happy to see this feature set added into ChatGPT – particularly when I'm looking for academic research in/on a subject I'm not familiar with.
The main hard part of searching isn’t formulating queries to write in the Google search bar, it’s clicking on links, and reading/skimming until you find the specific answer you want.
Getting one sentence direct answers is a much superior UX compared to getting 10 links you have to read through yourself.
Google does offer an AI summary for factual searches and I ignore it as it often hallucinates. Perplexity has the same problem. OpenAI would need to solve that for this to be truly useful
If we assume that people want a 'direct answer', then of course a direct answer is better. But maybe some of us don't want a 'direct answer'? I want to know who's saying what, and in which context, so I can draw my own conclusions.
- I count calories... eat out always and at somewhat healthy chains (Cava, Chipolte, etc). Tell GPT (via voice while driving to & or after eating) what ive eaten half the day at those places and then later for dinner. It calculates a calorie count estimation for half the day and then later at dinner the remaining. I have checked to see if GPT is getting the right calories for things off websites and it has.
- Have hiking friends who live an hour or two hours away and we hike once a month an hour or less drive is where we meet up and hike at a new place. GPT suggests such hikes and quickly (use to take many searches on Google to do such). Our drives to these new hikes learned from GPT have always been under an hour.
So far the information with those examples has been accurate. Always enjoy hearing how others use LLMs... what research are you getting done in one or two queries which used to take MANY google searches?
Case in point: Visual Basic for Applications (the Excel macro language). This language has a broad pool of reference material and of Stack Overflow answers. It doesnt have a lot of good explicatory material because the early 2000s Internet material is aging out, being deleted as people retire or lose interest, etc.
(To be frank, Microsoft would like nothing more than to kill this off completely, but VBA exists and is insanely more powerful than the current alternatives, so it lives on.)
1) I know a little bit about something, but I need to be able to look up the knowledge tree for more context: `What are the opposing viewpoints to Adam Smith's thesis on economics?` `Describe the different categories of compilers.`
2) I have a very specific search in mind but it's in a domain that has a lot of specific terminology that doesn't surface easily in a google search unless you use that specific terminology: `Name the different kinds of music chords and explain each one.`
LLMs are great when a search engine would only surface knowledge that's either too general or too specific and the search engine can't tell the semantic difference between the two.
Sometimes when I'm searching I need to be able to search at different levels of understanding to move forward.
> "Can you provide a list of the ten most important recent publications related to high-temperature helium-cooled pebble-bed reactors and the specific characteristics of their graphite pebble fuel which address past problems in fuel disintegration and dust generation?"
These were more focused and relevant results than a Google Scholar keyword-style search.
However, it did rather poorly when asked for direct links to the documentation for a set of Python libraries. Gave some junk links or just failed entirely in 3/4 of the cases.
But I'm not keeping my hopes up, I doubt the model has been explicitly fine-tuned to double check its embedded knowledge of these types of facts, and conversely it probably hasn't even been successfully fine-tuned to only search when it truly doesn't know something (i.e. it will probably search in cases where it could've just answered without the search). At least the behavior I'm seeing now from some 15 minutes of testing indicates this, but time will tell.
I consider myself quite anti LLM hype and I have to admit it has been working amazingly good for me.
I feel like there is a mental architecture to searching where you try and isolate aspects of what you are searching for that are distinct within the broad category of similar but irrelevant things. That kind of mental model I would hope still works well.
For instance consider this query.
"Which clothing outlets on AliExpress are most recommended in forum discussions for providing high quality cloths, favour discussions where there is active engagement between multiple people."
OpenAI search produces a list of candidate stores from this query. Are the results any good? It's going to be quite hard to tell for a while. I know searching for information like this on Google is close to worthless due to SEO pollution.
It's possible that we have at least a brief golden-age of search where the rules have changed sufficiently that attempts to game the system are mitigated. It will be a hard fought battle to see if AI Search can filter out people trying to game AI search.
I think we will need laws to say AI advice should be subject to similar constraints as legal, medical, and financial advice where there is an obligation to act in the interests of the person being advised. I don't want to have AI search delivering the results of the highest bidder.
When it starts with this you KNOW it's going to be maximum bad faith horsefuckery in the rest of the "question."
Which is also exactly something a bad-faith commenter would say, but if I lose either way, I'd rather just ask the question ¯\_(ツ)_/¯
my search skills are good but either way that requires 3+ searches and visiting the menu of each restaurant and checking their hours and reservation. remains to be seen if chatgpt search is consistently good at this though
This becomes even better if the information you want is in multiple different places. The canonical question for that used to be "what was the phase of the moon when John Lennon was shot?". There didn't used to be an answer to this in Google - but the AI search was able to break it down, find the date John Lennon was shot (easily available on Google), find the moon phase on that day (again, easily available on Google) and put them together to produce the new answer.
For a more tech relevant example, "what is the smallest AWS EC2 I can run a Tomcat server in?
You 100% can get this information yourself. It just much more time than having an AI do it.
The LLMs are nice because they are not yet enshitified to the point of uselessness.
Dead Comment
If you don't like that (like I do), you can also manually add it under Site Search using
https://chatgpt.com/?q=%s&hints=search
I can definitely see this new search feature being useful though. The old one was already useful because (if you asked) you could have it visit each result and pull some data out for you and integrate it all together, faster than you could do the same manually.
It's often hobbled by robots.txt forbidding it to visit pages, though. What I really want is for it to use my browser to visit the pages instead of doing it server side, so it can use my logged-in accounts and ignore robots.txt.
Deleted Comment
Deleted Comment
They really had the potential to do something interesting, but were just focused on their ad metrics with the "good enough" search box. What have they been doing all the time?
[0] https://static1.makeuseofimages.com/wordpress/wp-content/upl...
https://www.newyorker.com/news/amy-davidson/tech-companies-s...
And, yet, aside from Aramco, they are the most profitable companies in the history of the world.
What does this mean? Like, I work there and I’d be pretty annoyed if they stopped turning a profit as a collapse of the stock price would affect my compensation.
It’s interesting to hear this take because I’m used to hearing the opposite: that Google is too focused on increasing short-term profit at the expense of product quality.
If they're collecting data it doesn't even work; I make no effort to hide from them and none of their ads are targeted to me. Meta, though, they're good at it.
For example `8 hours ago: "autohotkey hotkeys"` with 4 links to pages which I visited while searching.
But this is a Chrome feature, not a Google Search feature. https://myactivity.google.com/myactivity does (sometimes? can't see it right now) have a grouping feature of all the searches made, but this is more of a search log than a search management feature.
So chrome://history/grouped is the closest to what I mean, but I can't pin or manage these history groups, enrich them with comments or even files, like pdf's which could then get stored in Google Drive, as well as get indexed for better searches.
I’m sure they will/are tackling this at the model level. Train them to both generate good completions while also embedding text with good performance at separating generated and human text.
The issue is with the query itself. You're assuming that there's some oracle that will understand your question and surface the relevant information for you. Most likely, it will use the word themselves as part of the query, which SEO sites will exploit.
A more pragmatic search workflow would be to just search for "most common programming languages used" [0], then used the Wikipedia page to get the relevant information [1]. Much more legwork, but with sources. And still quite fast.
[0]: (Screenshot) https://ibb.co/ggBLy8G
[1]: (Screenshot) https://ibb.co/H4g5bDf
is this a real question you needed an answer to, or a hypothetical you posed to test the quality of search results?
of course you're going to get listicles for a query like that, because it sounds like a query specifically chosen to find low-quality listicles.
I got:
This is pretty bad. (???!!! added by me)However, my follow up query "Provide primary web page for each language listed above" was quite decent:
Here are the primary websites for the programming languages mentioned:
The problem was with the 3rd query "Provide latest version for each language as mentioned on the primary website for that language."That brought back the first result basically unchanged.
So certainly this is a work in progress but very promising.
Deleted Comment
Deleted Comment
I'd assume right now the SEO target is still mainly Google rather than ChatGPT, but that's only an "I recon" not a citation.
If and when ChatGPT does become the main target for SEO spam, then Googling may start giving good results again.
As of October 31, 2024, the latest version of Java is Java 23, released on September 17, 2024. The most recent Long-Term Support (LTS) version is Java 21, released on September 19, 2023.
Which all seems correct and accurate.