Risks and Biases of AI in Scientific Content Generation

The phenomenon known as "hallucination" in artificial intelligence — content generation that sounds plausible but is wholly or partially invented — has different consequences depending on the domain in which it occurs. In generic marketing writing, a hallucination is a correctable error. In medical-scientific content, it can become a patient safety risk or a direct regulatory risk for the organization that publishes it.

A conceptual analysis identifies the technical origin of the problem: language models learn from the data they were trained on, and when that data contains historical, social, or sampling biases, the systems not only replicate them but often amplify them in their generated content ^[1]. In scientific contexts, this translates into invented studies, inaccurate references, or claims without real support — a serious risk to the integrity of the literature if not caught during review ^[1].

It's important to distinguish two types of bias that tend to get mixed up in public conversation about AI: training-data bias (which populations, languages, and sources are over- or under-represented) and factual hallucination (invented content that appears truthful). Both require different mitigations: the first calls for diversity and representativeness in the sources consulted; the second calls for mandatory cross-verification of every factual claim against primary sources.

For a team producing scientific content with AI assistance, the most effective mitigation documented so far isn't technical — it's procedural: no AI-generated content should be published without a human expert having verified every citation, every numerical figure, and every clinical claim against the original source.