Catching AI hallucinations

Catching AI Hallucinations: Why Cross-Checking Works Best When the Checks Are Less Correlated

AI systems can produce remarkably useful work, but they can also generate information that is false, misleading, outdated, unsupported, or entirely invented. The most dangerous errors are often not obvious nonsense. They are plausible statements presented confidently and surrounded by accurate information.

One of the best ways to reduce this problem is to cross-check important work with other AI systems. But there is an important qualification: asking multiple AIs does not automatically mean that you have obtained multiple independent checks.

If several systems reach the same conclusion because they rely on closely related underlying models, overlapping training data, similar reasoning patterns, or the same widely repeated misinformation, agreement between them may be less reassuring than it appears.

Agreement is a signal, not proof.

That single idea is the spine of everything below.

Why Multiple AIs Can Still Make the Same Mistake

Suppose you ask one AI to write an article and then ask another AI, based on closely related technology, whether the article is accurate. The second AI may catch errors, contradictions, or weak reasoning. This is still useful.

However, it is not the same thing as obtaining a check whose errors are unlikely to overlap with the first system's errors.

Closely related AI systems may share overlapping:

• training data,
• model architectures or development techniques,
• post-training and alignment methods,
• reasoning habits,
• common misconceptions, and
• blind spots.

This does not mean that related systems will necessarily make the same mistakes. Different prompting, retrieval, tuning, and system design can produce very different answers. It simply means agreement between them should not automatically be treated as strong confirmation.

This intuition has some grounding in ensemble-learning research, which generally holds that a group of models is most useful not simply because it contains multiple models, but because its members make errors that are not highly correlated with one another. That research was developed mostly for classifiers and regressors rather than language models, so treating it as direct proof about AI chatbots would overstate the case — but as a general principle, it supports the idea that diversity helps mainly when it reduces the chance that every checker fails in the same way.

It's also worth being honest about a limit here: frontier AI models increasingly draw on similar web-scale training data, similar reinforcement-learning-from-human-feedback techniques, and similar safety fine-tuning approaches. As the major labs converge on similar methods, "different company" may be a weaker proxy for "genuinely different errors" than it was a few years ago. Family diversity is still worth pursuing — it's just not as strong a guarantee as it may sound.

The Copilot Example: A Product, Not Just a Model

Consider Microsoft Copilot and ChatGPT. Copilot has historically incorporated OpenAI technology, although the exact underlying systems and product configurations can change over time. Just as importantly, Copilot is a product ecosystem: depending on the specific Copilot experience, its answers can be grounded in an organization's own data, permissions, and retrieval systems, not just a single base model. That means how much a Copilot answer actually overlaps with an OpenAI ChatGPT answer can vary by product, configuration, and what data the system was allowed to draw on.

Therefore, having ChatGPT review something produced by Copilot is not the same as asking one system to check itself. But it is also not a clean, fully independent check — the honest answer is "somewhere in between," and where exactly depends on details most users can't easily see.

Family Diversity: A Practical Heuristic, Not a Measurement

The industry shares data sources, research ideas, benchmarks, and increasingly other forms of technological influence. No two AI products are ever going to be as cleanly "independent" as two separately conducted scientific experiments.

So rather than claiming independence, it's more accurate to say this:

Multiple AI systems are most useful as cross-checks when their errors are less likely to be correlated. "Family diversity" is simply a practical shorthand for trying to create that condition — it is not a formal measurement of statistical independence.

Used this way, family diversity becomes a useful default habit rather than a guarantee: when an important claim has already been evaluated by one AI system, checking it with a substantially different system may increase the chance of catching an error that the first system missed. It's a way of stacking the odds, not a proof of correctness.

Six Kinds of AI Errors, and How to Catch Each One

Not all hallucinations look alike, and they don't all call for the same check. Below is a practical, plain-language breakdown of common failure types. These labels are informal working categories, not a standardized scientific taxonomy — but they're specific enough to be useful when deciding where to spend your review effort.

Failure Type	What It Looks Like	Best Check
Factual error	A date, number, event, quote, or attribution is simply false	Check against an authoritative source
Source-grounding error	A citation exists but doesn't actually support the claim attached to it	Open the source and test claim against text
Context error	A source is quoted accurately but key qualifiers or limitations are dropped	Read the surrounding source context, not just the quote
Reasoning error	The evidence may be true, but the conclusion doesn't follow from it	Ask for assumptions and counterarguments explicitly
Retrieval error	The system pulled irrelevant, outdated, or low-quality material	Evaluate source quality, relevance, and date
Temporal error	A once-accurate statement is now outdated	Confirm current status with current sources

Diversity is most worth the extra effort against the first four rows — the errors that involve knowledge, judgment, or interpretation. Mechanical issues like formatting and grammar don't need a diverse panel of checkers; almost any capable AI handles those fine on its own.

Model Diversity Is Not the Same as Evidence Diversity

Suppose one AI checks an article using only its learned knowledge, while another searches the web, retrieves current sources, and provides citations. Even if the underlying models were similar, the two systems would still be operating with different information pipelines. That's a second, distinct kind of diversity: evidence diversity.

An AI that searches for current information is not relying entirely on what was encoded during training. This is one reason search- and retrieval-oriented systems such as Perplexity can be particularly useful in a verification pipeline.

However, retrieval does not eliminate hallucinations. A retrieval-based AI can still find the wrong sources, select weak or unreliable ones, misread a source, cite a source that doesn't actually support the claim, combine information incorrectly, or draw conclusions that go beyond the evidence. Research evaluating AI legal-research tools has found that even citation-bearing, retrieval-grounded systems can produce answers with citations that don't fully support the claims attached to them — the source is real, but it doesn't say what the AI says it says.

Memory-based AI can hallucinate from what it thinks it knows. Retrieval-based AI can hallucinate, misinterpret, or overstate what it finds.

Both are valuable precisely because their failure modes are different — which is itself a form of diversity worth deliberately combining. (In the interest of practicing what this article preaches: the ensemble-learning and legal-research findings referenced above and earlier in this piece are described from general recollection and have not been independently re-verified against the original papers as of this writing. Treat them as directionally credible rather than precisely quoted, and verify directly before relying on them for anything consequential.)

Perplexity and Other Search-Based Systems as Fact Checkers

A search-oriented AI is particularly useful for checking factual claims, recent events, citations, statistics, or statements that may have changed since a model's training cutoff. Its advantage isn't primarily that it belongs to a different model family — it's that it can actively look for evidence.

For example, imagine Claude produces the statement: "Organization X adopted Policy Y in 2024." A second AI relying mainly on learned knowledge might agree — mildly reassuring, but it doesn't establish truth. A search-oriented system can instead attempt to locate the actual announcement, filing, or policy document. That's a meaningfully stronger form of verification.

But the process shouldn't stop once a citation appears. The important question is always: does the cited source actually say what the AI claims it says? For important claims, open the source and check it yourself.

Three Kinds of Diversity

1. Model Diversity

Use substantially different AI model families when practical. The goal isn't a perfectly independent vote — that may not exist in an industry with heavy overlap in data, methods, and research. The goal is increasing the odds that one system catches what another missed.

2. Evidence Diversity

Use systems that obtain information in different ways: one working from learned knowledge, another searching the live web, another checking a specific source or document, and the human researcher examining the original evidence directly. This can matter more than simply asking several systems the identical question.

3. Method Diversity

Don't ask every AI to do the same job. If four AIs are all asked "Is this accurate?", they tend to converge on similarly reassuring answers. Assigning different jobs is stronger:

• AI 1: Write or improve the article.
• AI 2: Look specifically for factual claims that may be false or unsupported.
• AI 3: Act as a hostile critic and search for weaknesses in the reasoning.
• AI 4: Verify whether citations actually support the claims attached to them.
• The human: Examine the most important original sources and make the final call.

Ask Adversarial Questions, Not Just "Is This Correct?"

One of the simplest ways to improve AI fact-checking is to change the question. Instead of:

"Is this article accurate?"

try:

Act as a skeptical fact-checker. Assume this article may contain plausible-sounding but false, misleading, outdated, or unsupported claims. Identify the five claims most likely to be wrong. Explain why each is questionable and what evidence would be needed to verify it.

Or:

Try to disprove the major claims in this article. Ignore grammar and style. Look for factual errors, unsupported assumptions, logical gaps, overgeneralizations, misleading wording, and conclusions that go beyond the available evidence.

Explicitly assigning the role of "look for failure" often matters as much as switching models entirely. It's worth noting one limit here too: asking a model to "show its reasoning" produces a persuasive-sounding explanation, not evidence. A fluent step-by-step justification for a wrong answer is still wrong. Reasoning prompts are good at surfacing unstated assumptions and gaps — they are not a substitute for checking claims against real sources.

A Practical AI Cross-Checking Workflow

The workflow below combines model, evidence, and method diversity. It's a starting framework to scale up or down, not a fixed formula — a casual post may need only a quick pass; a piece with load-bearing factual claims justifies more.

Stage	Primary Goal	Best Type of Check
Drafting	Create the initial work	Any capable AI appropriate to the task
Logical review	Find contradictions and weak reasoning	A different AI with a skeptical prompt
Factual review	Identify questionable factual claims	A less-correlated model family when practical
Evidence review	Locate supporting or contradictory evidence	Search- or retrieval-based AI
Citation review	Confirm sources support the claims	Open and inspect the original sources
Final review	Make the final judgment	Human judgment informed by the evidence

For medical, legal, financial, scientific, or other high-stakes material, this workflow should feed into direct consultation of authoritative sources and, where appropriate, qualified human experts — not substitute for them.

A Rough, Conditional Hierarchy

Not every form of cross-checking provides the same confidence, but the order below is a rough starting intuition, not a strict ranking — the quality of a check's task design and evidence often matters more than which numbered rung it occupies:

1. The same AI reviews its own answer.
2. The same AI is asked again with a different, more adversarial prompt.
3. A different implementation using related technology reviews the answer.
4. A substantially different AI model family reviews the answer.
5. A search- or retrieval-based system checks factual claims against current sources.
6. The cited primary or authoritative sources are opened and examined directly.
7. For high-stakes matters, qualified human expertise is added.

Treat this as a risk-based ladder, not a scale of guaranteed reliability. A highly capable AI can sometimes catch an error that a "more independent" AI misses, a different model family can just as easily repeat a popular false claim from the open web, and a retrieval system can fail badly if it finds or misreads poor sources. The list is a useful default for deciding where to spend limited review time — nothing more.

A Note on AI Providers

Anthropic (Claude), OpenAI (GPT), Google (Gemini), Microsoft (Copilot), Mistral, xAI (Grok), DeepSeek, Alibaba (Qwen), Meta (Llama), Amazon (Nova), and Cohere all represent different organizations, but company name alone is a weak proxy for how differently two systems will actually fail. Some of these are closed products built on shared or licensed underlying models; others are open-weight and independently fine-tuned by many different groups. The point of naming them isn't to suggest that picking one from each column guarantees independence — it's a reminder that a verification pipeline doesn't have to consist entirely of closely related systems.

The Human Still Has the Final Responsibility

Multiple AIs agreeing with one another does not transform a statement into a fact. AI systems can reinforce one another's errors: one invents a claim, another judges it plausible, a third rewrites it more convincingly. A polished consensus can still be wrong.

It's worth naming why that happens on the human side, too. A well-organized, confidently written answer feels more trustworthy than it has actually earned — regardless of whether one AI or five agree with it. Treating several similar-sounding answers as independent confirmation, when they may share the same data, tools, or blind spots, is its own quiet failure mode. The fix isn't complicated, just easy to skip under time pressure: ask what was actually checked, not just how many systems agreed.

The strongest defense isn't blind faith in whichever AI sounds most confident. It's a process: use different systems, give them different tasks, look for disagreement, search for evidence, check important sources directly, and make the final judgment yourself.

The better questions to keep asking are not "do several AIs agree?" but rather: how likely is it that these checks would fail in the same way? and what evidence would show that all of them are wrong? Those questions turn AI cross-checking from a collection of reassuring opinions into a real system for catching mistakes.