The proliferation of digital and social media technologies has enabled quick and wide dissemination of news stories and press releases about new medical treatments. Evaluating these stories is difficult for two reasons. First, these stories are often not completely true or false. A nuanced approach that considers different aspects of these stories (e.g., the presence of inflated claims, suppression of risks associated with the treatment or withholding other essential information) is more appropriate for evaluation. Second, evaluating the quality and completeness of the arguments in the stories is costly and requires expertise in the relevant medical field, which laypeople do not have. To address this problem, in this study, we train different machine learning models on multi-criteria expert evaluations for health news stories about new medical treatments and compare their performance. We then compare the machine learning model evaluations to laypeople evaluations. We find that machine learning models overall outperform laypeople, who have a propensity to overestimate the comprehensiveness of the claims. Our machine learning models employ multi-criteria evaluation, which is different from most previous studies that evaluate news stories on whether they are true or false. We conclude by discussing the implications of this study for consumers of health news stories disseminated via social media.

中文翻译：

---

健康新闻报道的多标准评估

数字和社交媒体技术的普及使得有关新疗法的新闻报道和新闻稿能够快速广泛地传播。由于两个原因，评估这些故事很困难。首先，这些故事往往并不完全正确或完全错误。考虑这些故事的不同方面（例如是否存在夸大的声明、抑制与治疗相关的风险或隐瞒其他重要信息）的细致入微的方法更适合评估。其次，评估故事中论点的质量和完整性成本很高，并且需要相关医学领域的专业知识，而外行人不具备这些专业知识。为了解决这个问题，在本研究中，我们训练了不同的机器学习模型，对有关新疗法的健康新闻报道进行多标准专家评估，并比较它们的性能。然后，我们将机器学习模型评估与外行评估进行比较。我们发现机器学习模型总体上优于外行人，外行人倾向于高估主张的全面性。我们的机器学习模型采用多标准评估，这与之前大多数评估新闻报道真假的研究不同。最后，我们讨论了这项研究对通过社交媒体传播的健康新闻故事的消费者的影响。