The Confident Wrong

Model collapse draws all the attention. The subtler failure is already here — systems that sound fluent while the facts underneath quietly disappear. The dangerous phase is not when AI breaks. It is when AI works perfectly and is wrong. A research team fed over eight hundred thousand medical data points through successive generations of AI models — clinical text, radiology reports, pathology images. They watched what happened as each generation trained on the output of the last. By the fourth generation, vocabulary had collapsed from 12,078 unique words to roughly 200 — a 98.9 percent reduction. Rare but critical pathologies — pneumothorax, pleural effusions — vanished from the generated reports entirely. Demographic representation skewed toward a single phenotype: middle-aged male. And the models kept issuing confident, well-formatted diagnostic reports throughout the entire process. The false reassurance rate tripled to forty percent. Blinded physicians who evaluated the output confi