Are You Hallucinating?

Recently, a publication printed an article about an exciting new RAB Research report. The problem is, we produced no such report. When talking to the editor, they indicated that they had relied on AI to help craft the article. 

There is no denying the benefits of AI as a tool, as a resource and as an assistant. But there is a downside. It’s not always correct. In the tech world, they call errors with AI “hallucinations.” Why they can’t just say it’s wrong is beyond me. 

Here is some verified research (with links) to indicate how prevalent this problem is. I did notice a trend that the more specialized the data, the higher the percentage of hallucinations. You’ll see that much of my sourcing for this research comes from medical websites.

There isn’t a single, universally agreed-upon “percentage” because hallucination rates vary widely across models, tasks, domains and evaluation methods – ranging anywhere from three to nine out of ten times on various LLMs. But multiple studies give us real data points:

1. Hallucination Rates Across Tasks & Models

• Some peer-reviewed studies found that GPT-3.5 hallucinated 39.6% of the time and GPT-4 hallucinated 28.6% of the time when producing accurate academic references, while another model accidentally fabricated 91% of citations. 
• Source:  National Library of Medicine
• Larger benchmarking studies show widely varying performance: In some legal-domain tests, models hallucinated across 50% to 80% of queries.
• Source: Healthcare IT News

  2. Dependency on Context

• What counts as a hallucination in one study (e.g., invented legal/medical facts) might be different in another (fabricated citations versus harmless inaccuracy). This means the risk is higher for fact-intensive professional output than casual use. 
• Source: Scientific Reports

3. Errors in News & Verification Contexts (This is what happened to us)

• A recent cross-platform analysis of AI assistants found that around 45% of news-related answers contained at least one significant error, and sourcing problems in roughly 72% of some models’ outputs. 
• Source: Reuters

The key takeaway:  Verification Isn’t Optional — It’s Essential

For public-facing information like news articles, academic claims, press releases, organizational statements, etc., you should:

1. Always double-check AI outputs against authoritative sources (original documents, people or trusted databases).
2. Insist on direct sourcing (e.g., “show me the primary source/URL/official publication”).
3. Use human reviewers with domain expertise, especially in high-impact or reputational contexts.
4. Deploy AI with retrieval-augmented systems that pull from verified data rather than relying solely on generative predictions. This is what we’ve done with our AI tools on RAB.com

AI can generate impressive text, graphics, video, etc., but can’t yet guarantee factual accuracy on its own. Research shows hallucinations — from minor inaccuracies to completely fabricated information — occur frequently enough in real-world use cases to require mandatory human verification and quality control before any public dissemination.

It can happen to all of us. This is one of the main reasons RAB has developed RAB-AI tools on RAB.com that ONLY look at our verified, accurate and trusted information. Our CopyWrite tool was written based on industry best practices for creative, and our Why Radio AI tool only looks at our research archives. This way, you know that when you use RAB’s AI tools, you are getting verified, accurate and trustworthy information.

Verify, verify, verify. The editor of the publication that made an error with our research was highly embarrassed and apologized profusely and will be printing a retraction. That hurts. You don’t want to put yourself in that position.

Think Big, Make Big Things Happen!

Jeff Schmidt is the SVP of Professional Development. You can reach him at Jeff.Schmidt@RAB.com. You can also connect with him on X, YouTube, and LinkedIn.