Stop me if you’ve heard this one: Google’s AI Overviews are sometimes wrong. This has been a perennial complaint about the Gemini-written summaries that appear at the top of search results since their debut in mid 2024.
It also happens to be true. Optimists might dismiss the infamous “glue on pizza” moment and others like it as simply early bugs in a new feature. But in the spring of 2026, The New York Times commissioned AI startup Oumi to study how frequently AI Overviews give bad answers. The latest version was accurate 91% of the time, which sounds pretty good until you consider Google’s billions of daily searches. At that scale, even a single-digit error rate can produce millions of inaccurate summaries every hour.
To hammer the point home in its write-up of the study, the Times pointed to BBC tech reporter Thomas Germain’s story about how he wrote up a fake blog post that declared himself the best hot dog-eating tech journalist on Earth. Within a day, Google’s AI Overviews were parroting that claim, apparently with little scrutiny.
{“blockType”:”mv-promo-block”,”data”:{“imageDesktopUrl”:”https://images.fastcompany.com/image/upload/f_webp,q_auto,c_fit/wp-cms-2/2025/03/media-copilot.png”,”imageMobileUrl”:”https://images.fastcompany.com/image/upload/f_webp,q_auto,c_fit/wp-cms-2/2025/03/fe289316-bc4f-44ef-96bf-148b3d8578c1_1440x1440.png”,”eyebrow”:””,”headline”:”u003Cstrongu003ESubscribe to The Media Copilotu003C/strongu003E”,”dek”:”Want more about how AI is changing media? Never miss an update from Pete Pachal by signing up for The Media Copilot. To learn more visit u003Ca href=u0022https://mediacopilot.substack.com/u0022u003Emediacopilot.substack.comu003C/au003E”,”subhed”:””,”description”:””,”ctaText”:”SIGN UP”,”ctaUrl”:”https://mediacopilot.substack.com/”,”theme”:{“bg”:”#f5f5f5″,”text”:”#000000″,”eyebrow”:”#9aa2aa”,”subhed”:”#ffffff”,”buttonBg”:”#000000″,”buttonHoverBg”:”#3b3f46″,”buttonText”:”#ffffff”},”imageDesktopId”:91453847,”imageMobileId”:91453848,”shareable”:false,”slug”:””,”wpCssClasses”:””}}
The stunt looks trivial because the query was trivial. But the underlying mechanism isn’t. While Germain’s experiment suggests AI Overviews are easy to manipulate, it succeeded mostly because it was the one article about a subject that no one had searched before—essentially an information vacuum. For a story where there’s a lot of coverage, a single random blog post likely wouldn’t have much effect.
Search, rewritten
However, the hot dog story shows only one way AI information systems in general—and AI Overviews in particular—can fail at their job: giving the user accurate information. And that accuracy matters more than ever: The presence of AI Overviews in searches has been growing considerably over the last year. An April report from AI-visibility startup QuickSEO pegged them as appearing in 60.23% of searches, and that was before Google used its May I/O developer conference to deepen the connection between AI Overviews and AI Mode, letting users move directly from a summary into a conversational follow-up.
The top of a search page isn’t the only place audiences encounter AI summaries, but it is the most important since it’s Google, the place everyone already goes to for search. Users of AI chatbots are deliberately seeking out AI help with their queries; Google users just get it. It may be inaccurate uncomfortably often, but publishers need to contend with the reality that their information will be represented through the lens of AI Overviews. And while they can’t control that lens, there is still opportunity—and responsibility—when their content passes through it.
The ubiquity of Overviews doesn’t mean people always take them as gospel. Trust in AI answers often varies with intent—or rather, the stakes of that intent. If you’re asking for a roast-chicken recipe, you’re probably less likely to scrutinize the answer than if you’re asking about cancer treatments. The starting point, however, is the same.
The user’s behavior may change with the stakes, but the initial framing has already happened. A user may question a consequential answer while still allowing it to supply the vocabulary, shape follow-up searches and determine what they investigate next. If a trusted publisher name is cited in the summary, it can increase confidence even when users never open the citation. I’ve written before about this in terms of value, but that value is dependent on being accurately represented, which publishers clearly have a stake in.
How accuracy breaks
To better understand how AI Overviews get things wrong, I spoke to Isis Blachez, the AI lead at Newsguard, in charge of the organization’s AI False Claims Monitor. She broke it down along three dimensions, which are reflected in the Times-commissioned study.
- The AI retrieves weak or irrelevant material. This is how a summary might end up recommending lathering a layer of Elmer’s glue onto your pizza to ensure your toppings stick. That was originally from a Reddit post that suggested it ironically (we hope!)—which made it irrelevant to the query.
The thing is, the original post did answer the question directly, which is a large factor in AI discoverability. While studies show AI engines do tend to favor journalistic content, if that content isn’t optimized for machines (or worse, blocked) then it can easily be overlooked, and something more weakly supported can end with an outsize share of the answer.
“We do [reliability] ratings of news sites,” explains Blachez. “And we saw that for most of the highly ranked sites, they were blocking a lot of the AI bots, and then most of the low-quality sources were giving full access to AI web crawlers.”
- The AI finds the correct source but misreads it. This might be the most important and least obvious way AI ends up making mistakes. Blachez points to a case in which multiple chatbots cited Snopes while confirming a false claim that Iran had attacked a Pakistani-flagged oil tanker. The Snopes article was actually debunking the claim rather than supporting it, but the AI got it backward.
“Sometimes, even if it’s citing a credible source, it can be incapable of citing it well or retrieving the information correctly,” Blachez says.
In these cases, there was nothing wrong with the underlying reporting. It’s the machine that got it wrong. This can be the most frustrating of AI failures, and it often factors in litigation over AI answers.
- The available information was deliberately manipulated. This is the hot dog situation. Writ large, it looks like the pro-Kremlin Pravda network, which flooded the web with millions of articles across sites designed to resemble news outlets, pushing Russian propaganda at machine scale.
While that’s an extreme example, it shows how coordinated actors who publish similar-sounding claims across many sites can create the appearance of consensus and dominate the material available to retrieval systems.
“So what we’ve observed that worked with Pravda is flooding search results,” says Blachez. “It’s like putting the same information with practically the same language, many domains, many times and just dominating narrative on that specific topic.”
Making the truth machine-readable
So there are several ways a “well-meaning” AI might be led astray through problems with access, manipulation, or the content itself. Stipulating that the operator of the AI engine has a responsibility to get good information to its users, what is the publisher’s responsibility here?
I think many journalists and publishers fall into a mentality of helplessness—thinking that there’s nothing they can do because they can’t control what an AI does with their content. But they can influence that across each of the three failure dimensions. To ensure their content is in the mix, it needs to be present (i.e. not blocked). To discourage misreads, it should be machine-optimized. And end-running manipulative posts means creating your own that directly answer the queries you want to compete.
Publishers have legitimate reasons to block crawlers: copyright concerns and the lack of compensation being the main ones. And if journalistic content is blocked, that doesn’t absolve Google and other AI services from scrutinizing and fact-checking the raw material for their answers. But if journalism is available to the AI, publishers do have levers they can pull to help give it the best chance of being correctly represented.
Most editorial operations already include a layer that optimizes for SEO. The best way to influence what people see in AI Overviews and chatbots when the work is surfaced is to add a machine-readability pass as well. This isn’t just standard GEO stuff like having titles that match certain queries. It means striving to ensure the difficult parts of a subject are clearer to an AI, even though they might already be clear to a human.
In practice, that means stating facts or conclusions that the prose otherwise leaves implicit. A human reader may understand that “alleged” qualifies an entire sequence of events, for example, even if the word appears only once. A machine may not carry that qualification forward.
Some questions editors might ask in this machine-readable pass:
- Are dates explicitly tied to the correct events?
- Is it clear whether an allegation is being reported, verified or debunked?
- Is the primary conclusion stated plainly rather than left entirely to implication?
- Are corrections and updates obvious?
- Does the article distinguish the original source from later repetition?
- Does the headline create ambiguity that the body later resolves?
As with SEO, adapting a piece for greater machine clarity often improves human clarity, too. But it only improves the odds; it can’t guarantee correct interpretation. The goal isn’t to make journalism “AI-proof.” It’s to eliminate avoidable ambiguity and give accurate information a better chance of surviving the answer layer.
Publishers can’t dictate what Google says about their work, and they shouldn’t be expected to fix the shortcomings of someone else’s product. But as AI becomes a default layer between journalism and its audience, helplessness isn’t much of a strategy. They can still make the truth easier to find, harder to misread and much more difficult to replace.
{“blockType”:”mv-promo-block”,”data”:{“imageDesktopUrl”:”https://images.fastcompany.com/image/upload/f_webp,q_auto,c_fit/wp-cms-2/2025/03/media-copilot.png”,”imageMobileUrl”:”https://images.fastcompany.com/image/upload/f_webp,q_auto,c_fit/wp-cms-2/2025/03/fe289316-bc4f-44ef-96bf-148b3d8578c1_1440x1440.png”,”eyebrow”:””,”headline”:”u003Cstrongu003ESubscribe to The Media Copilotu003C/strongu003E”,”dek”:”Want more about how AI is changing media? Never miss an update from Pete Pachal by signing up for The Media Copilot. To learn more visit u003Ca href=u0022https://mediacopilot.substack.com/u0022u003Emediacopilot.substack.comu003C/au003E”,”subhed”:””,”description”:””,”ctaText”:”SIGN UP”,”ctaUrl”:”https://mediacopilot.substack.com/”,”theme”:{“bg”:”#f5f5f5″,”text”:”#000000″,”eyebrow”:”#9aa2aa”,”subhed”:”#ffffff”,”buttonBg”:”#000000″,”buttonHoverBg”:”#3b3f46″,”buttonText”:”#ffffff”},”imageDesktopId”:91453847,”imageMobileId”:91453848,”shareable”:false,”slug”:””,”wpCssClasses”:””}}
Source link
