“Spring Health serves as a model for how AI can be harnessed to Make America Healthy Again… Spring recommends that to avoid risks to health and mitigate harm, generative AI-enabled mental health tools should be evaluated, and the evaluation should be:

• Clinically Informed. Experienced clinicians should be included at every stage of the design and validation process.
• Narrowly Scoped. Because safety in mental health is difficult to quantify, the evaluation should focus on clear, well-defined concerns.
• Multi-turn. Single-turn conversation evaluations, which involve sending a single prompt to the AI system and assessing its response, are not enough to evaluate for clinical safety. Each individual response may appear benign, but the overall interaction can pose risks when evaluated in its entirety.
• Automated. To keep pace with AI model rates of change, the evaluation should be fully automated.
• Model Agnostic. The evaluation should be agnostic of the specific AI system. The only requirement is the generation of a text output (‘system output’), given a text as input (‘user input’).
• Multi-metric. Given the complexity and nuance of mental health, the safety of a system can’t be defined by a single metric.”

“Slingshot AI continues to believe that general wellness apps like Ash can provide enormous benefit at low risk by simply instituting basic guardrails and transparency. In the future, these apps may be further refined and rise to a level that is more consistent with treatment and diagnosis. When that time comes, we encourage the DHAC to recommend that the FDA consider instituting a more flexible, contemporary risk-aligned regulatory framework that accounts for rapidly evolving technologies, including generative AI. In the meantime, we would urge the DHAC to encourage the FDA to formalize a dedicated ‘Wellness Guidance’ for GenAI-enabled digital mental health apps like Ash. To be clear, this means a dedicated FDA wellness designation with simple guardrails and transparency that promotes resilience, daily support, and wellbeing, that is free to provide its services to the American people without the disproportionately burdensome requirement of high-risk applications.”

“Proposed ‘Example of Software Functions that are the focus of FDA’s regulatory oversight (Device Software Functions and Mobile Medical Apps)’

• A software function that uses Gen AI to engage in personalized dialogue to diagnose, assess, or create/manage individualized mental health treatment, or representing itself as a licensed professional or “therapist.” These functions have a device intended use by nature, whether by directly providing diagnosis or treatment, or implying it by presenting in dialogue as a therapist or healthcare agent (virtual or otherwise).”

“This final rule establishes the conditions under which a prescription only drug product can be used by consumers without the supervision of a practitioner licensed by law to administer such drug when an additional condition to ensure appropriate self-selection or appropriate actual use, or both, is implemented. OPH asks the Committee to recommend that Center for Devices and Radiological Health (CDRH) consider whether [a similar rule] for medical devices broadly, or, at minimum for SaMD, would be a viable pathway to promote greater access to nonprescription SaMD for certain conditions. We encourage the Committee to consider ways in which validated, patient-facing software-based screening tools can be leveraged to effectively aid patients in understanding their condition and available treatment options. Leveraging these digital screening tools as a part of the product or the patient information may help to ensure appropriate use of the tools in appropriate non-prescription contexts. Such an option could significantly expand access to these clinically validated, FDA-vetted technologies in a way that allows for scalable implementation independent of individual HCP participation.”

“Broadly, GenAI has significant potential to support cleared DTx products in helpful but non-therapeutic ways. Patient onboarding, ‘nudges’ or reminders to use the product, or more personalized interactions leading to improved engagement, could all be safely and effectively implemented without altering or compromising the underlying therapeutic content. To the extent that GenAI is introduced into therapeutic content, we believe that effectiveness must be established through high quality clinical research. We see inherent challenges in constructing robust scientific evidence validating an open model GenAI tool. Because each interaction will be unique, there is no guarantee that one individual patient experience will mimic another. There is an opportunity here to use validated reporting tools at scale to obtain large quantities of real world evidence, to minimize any statistical error in analyzing the data and drawing conclusions about effectiveness for these more variable interventions. In addition to unique challenges with evidence generation, there are unique challenges for risk mitigation in GenAI-based therapeutics. New mitigations will likely need to be implemented to address known product risks. We do not, however, see GenAI presenting a new safety risk relative to the patient’s pre-existing condition(s).”

• “The FDA should establish a clear framework for evaluating the safety and effectiveness of generative AI-enabled mental health devices, including the use of validated psychiatric assessment tools and ongoing post-market surveillance to monitor evolving model behavior.
• That the FDA requires evidence demonstrating that AI-enabled systems function fairly across demographic, cultural, and diagnostic subgroups.
• The adoption of standardized labeling framework for AI-enabled devices that discloses the data sources used for training and validation, the model’s intended scope, capabilities, and limitations, and known uncertainty levels or confidence intervals for outputs.
• Regulatory standards specifying whether patient data are processed locally or via cloud services. Secondary data use, such as for marketing or retraining, should be prohibited without explicit, informed consent.
• The FDA shift liability to developers from clinicians and health systems to better balance the risks and duty of care of a tool that is being implemented in clinical care.”

“The field of mental health AI is diverse–ranging from self-help chatbots to clinical decision support–so a rigid, one-size-fits-all validation approach could inadvertently hamper beneficial technologies.

• Flexible Validation: FDA should allow a variety of evidence to demonstrate an AI tool’s safety and effectiveness. In some cases traditional trials may be feasible; in others, real-world evidence or observational studies might suffice. The key is to focus on meaningful outcomes and performance standards, rather than prescribing one uniform testing method for all products. FDA can provide guidance on best practices, but developers should have leeway to choose validation methods suited to their tool and patient population, as long as the evidence is sound.
• Proprietary Information: We also urge caution on any requirements to publicly disclose sensitive proprietary details, such as source code or algorithms. Regulators should have access to detailed information as needed for evaluation, but this can be done confidentially. Forcing companies to reveal trade secrets or extensive training data is not necessary for accountability and would discourage innovation. We support transparency about performance–for example sharing summary results, intended use, and limitations–without requiring companies to expose their intellectual property. This balanced approach to keeping proprietary details confidential while being transparent with performance data protects both patients and the incentives to develop new mental health technologies.”

Links to E&O’s reports, databases, newsletters