Quality and safety of artificial intelligence generated health information

AI errors

Across all types of AI, errors are a common challenge. As the text, audio, and video output of modern generative AI has become increasingly sophisticated, erroneous or misleading responses may be difficult to detect. The phenomenon of “AI hallucination” has gained prominence with the widespread use of AI chatbots (eg, ChatGPT) powered by LLMs. In the health information context, AI hallucinations are particularly concerning because individuals may receive incorrect or misleading health information from LLMs that are presented as fact.23 For members of the general public, who may lack the capability to distinguish between correct and incorrect information, this has considerable potential for harm. For healthcare professionals using LLMs to generate clinical documentation, the generated outputs must be treated as drafts that require careful review for accuracy before finalisation.

Numerous technological strategies are being explored to minimise potential risks associated with generative AI errors. One promising strategy for accurately answering health related questions involves developing generative AI applications that “ground” themselves in relevant sources of information. This approach diverges from earlier methods that relied on responses being generated from model “memory.” Instead, many AI applications can now access and subsequently summarise information from up-to-date, authoritative sources. For example, many AI chatbots now incorporate real time internet search capabilities to return responses that summarise and explicitly cite the information source. Another approach is to improve “uncertainty quantification” for generative AI. This involves developing generative AI that better communicates the level of uncertainty associated with its response. Therefore, when unsure about an answer, the response should clearly highlight the uncertainty, thereby allowing the user to interpret the information more appropriately.

Health disinformation

As distinct from AI hallucinations, where incorrect or misleading information is generated inadvertently, it is also possible for malicious actors to intentionally generate incorrect or misleading information using generative AI if effective guardrails are lacking. When incorrect or misleading information is generated deliberately, it is referred to as disinformation. Although disinformation is not new, generative AI may enable the inexpensive creation of diverse, high quality, targeted disinformation at scale.45 This problem is not specific to health, but the effects of enhanced health disinformation are likely to be particularly problematic for society.

One option for preventing AI generated health disinformation involves fine tuning models to align with human values and preferences, including avoiding known harmful or disinformation responses from being generated. An alternative is to build a specialised model (separate from the generative AI model) to detect inappropriate or harmful requests and responses. The specialised model would screen the request before allowing it to be passed to the generative AI model, and the output of the generative AI model would be screened before releasing the output. In the linked study (doi:10.1136/bmj-2023-078538), we highlighted that many popular AI chatbots and assistants—including ChatGPT, Copilot, Bard, and HuggingChat—lack effective guardrails for preventing the generation of health disinformation.4

Initiatives to facilitate the easy identification of AI generated content, such as embedding digital watermarks, are also underway. Progress, however, is still required towards industry standards for identifiable AI generated material. Such efforts would make it easier for content sharing platforms (eg, social media, search engines) to identify and remove inappropriate AI generated content.6

Equally critical to countering AI facilitated disinformation is the establishment of robust AI vigilance processes. As generative AI continues to develop, emergent and unforeseen risks are likely to arise, underscoring the importance of ongoing monitoring, fixing identified safeguard vulnerabilities, and transparency. Our study found a lack of transparency among generative AI developers regarding the safeguards and processes implemented to minimise risks from health disinformation, along with a deficiency in responding to and fixing reported vulnerabilities related to health disinformation.4

Privacy and bias

The privacy of personal health information must be prioritised during the development and use of generative AI.7 Private health information should not be used to train generative AI models, as it is difficult to ensure that sensitive information will not leak into model outputs. Healthcare professionals need to also carefully consider the consequences of inputting sensitive patient information into public AI assistants and chatbots for tasks such as drafting clinical summaries, communications, and emails. Generative AI applications often state terms and conditions that allow developers to store and use information entered. The public should also be aware of this to avoid inputting sensitive information. Therefore, for sensitive data, it is important to only use generative AI services that explicitly commit to not retaining data, or to run the generative AI model locally to ensure that health data are not sent to a third party.

Training of generative AI requires vast amounts of text, image, and audio content, often sourced from the internet. In learning from this diverse material, the AI model is at risk of inheriting biases present in the training material, and hence deployment of the AI risks reinforcing existing inequities.78 Despite efforts by developers to mitigate biases, it remains challenging to fully identify and understand the biases of accessible LLMs owing to a lack of transparency about the training data and process.8 Ultimately, strategies aimed at minimising these risks include exercising greater discretion in the selection of training data, thorough auditing of generative AI outputs, and taking corrective steps to minimise biases identified.

Concluding remarks

A fundamental current challenge is the rapid progress of AI. One consequence of the frequent release of new AI models, or updates to existing AI models, is that performance and associated risks may change rapidly. For example, in our study, Microsoft’s Copilot demonstrated effective safeguards preventing the generation of health disinformation in September 2023, but three months later these safeguards were no longer present.4 Such a finding outlines that frequent ongoing audits of risks and functionalities will be required.

For readers seeking a deeper understanding of AI safety and ethics as it relates to health, we refer to World Health Organization guidance on ethics and governance of AI for health,9 and the European Parliamentary Research Service report on the applications, risks, ethics, and societal impacts of AI in healthcare.10 These documents provide valuable insights into responsible deployment and management of AI technologies, emphasising the critical need for ongoing auditing and adaptation in this rapidly evolving specialty.