Artificial intelligence (AI)-enabled chatbots are increasingly being used to help people manage their mental health. Chatbots for mental health and particularly ‘wellness’ applications currently exist in a regulatory ‘gray area’. Indeed, most generative AI-powered wellness apps will not be reviewed by health regulators. However, recent findings suggest that users of these apps sometimes use them to share mental health problems and even to seek support during crises, and that the apps sometimes respond in a manner that increases the risk of harm to the user, a challenge that the current US regulatory structure is not well equipped to address. In this Perspective, we discuss the regulatory landscape and potential health risks of AI-enabled wellness apps. Although we focus on the United States, there are similar challenges for regulators across the globe. We discuss the problems that arise when AI-based wellness apps cross into medical territory and the implications for app developers and regulatory bodies, and we outline outstanding priorities for the field.

支持人工智能 (AI) 的聊天机器人越来越多地被用来帮助人们管理心理健康。用于心理健康，特别是“健康”应用的聊天机器人目前存在于监管“灰色地带”。事实上，大多数由人工智能驱动的生成式健康应用程序都不会受到健康监管机构的审查。然而，最近的研究结果表明，这些应用程序的用户有时会使用它们来分享心理健康问题，甚至在危机期间寻求支持，并且这些应用程序有时会以增加用户受到伤害的风险的方式做出反应，这是当前的挑战美国的监管结构没有足够的能力来解决这个问题。在本视角中，我们讨论了人工智能健康应用程序的监管环境和潜在健康风险。尽管我们关注的是美国，但全球监管机构也面临着类似的挑战。我们讨论了基于人工智能的健康应用程序进入医疗领域时出现的问题以及对应用程序开发人员和监管机构的影响，并概述了该领域的突出优先事项。