AI models providing health information must operate within a regulatory framework, expert says

Ahead of a London event on May 12 on how AI is rapidly reshaping how healthcare information is discovered, interpreted and shared, this week's Expert View comes from panellist Sophie Randall, Director of the Patient Information Forum, the UK membership organization for people working in health information and support.  

AI has revolutionized the way people seek and consume health information. The technology has been deployed at pace, public behaviour change is entrenched and organizations are struggling to understand how to reconnect with their audiences.

But AI platforms have been released in a highly competitive environment. The tools are ‘learning on the job’ and information quality and patient safety has been compromised.

Nine in 10 people go online to seek information about their health. Our data shows only one in 10 adults with long-term conditions are signposted to health charities. Google traditionally filled this void, connecting people with information and support. It was the most used source for health advice alongside the NHS website in 2024. ¹

Concerns over inaccurate content

The launch of Google AI Overviews (AIO) in 2024 created barriers for people seeking health informateion. Six in 10 health charities report a fall in web traffic since the launch of APIs and some a drop in calls to helplines. ² The Pew Centre found only one in 10 people click beyond the AOI, which neatly fills a smart phone screen. ³

Google’s CEO said people should not rely on AIOs alone. ⁴ This is good advice. Health charities found inaccuracies, misinformation and hallucinations in AIOs. ⁵

PIF and a coalition of 70 health charities, including the British Heart Foundation, Cancer Research UK, Macmillan and Marie Curie presented a dossier of evidence containing screenshots of inaccurate content to Google in May 2025. It identified key concerns:

1. Risks to health outcomes

2. Quality and accuracy of content

3. Lack of localized source material

4. Impact on health inequalities

5. Access to support

We asked Google to:

• Agree a framework for the verification of trusted health information
• Ensure results prioritise verified UK sources
• Provide clearly visible context and support options at every stage
• Introduce explicit warnings that summaries may not be accurate
• Provide directions on seeking medical advice

Google only acted to remove potentially dangerous content following The Guardian ⁶ investigation based on our evidence in January 2026.

This summer our coalition is updating and reissuing our call for action. It will extend to other large language models like ChatGPT. Usage of the platform doubled in between 2024-25.

Safeguards are important because public behaviour change is rapid. Two in 10 use Google AI summaries and ChatGPT for health advice. And six in 10 regular ChatGPT users say they ask questions about their health and wellbeing. ⁷

AI platforms overaking social media

A survey of more than 2000 UK women, commissioned by The Eve Appeal (YouGov, March 2026 ⁸ ) for its annual Get Lippy campaign, found AI platforms had overtaken social media as a place women go to with their gynae health questions.

The charity ran 25 test scenarios using ChatGPT. Members of staff posed as patients with ‘red flag’ symptoms of gynaecological cancers. ChatGPT failed to identify ovarian and vulval cancer as possible causes of their symptoms.

Mental health charity MIND is so concerned about the impact of the models on mental health, it has launched a year-long commission on the topic. ⁸

Data from Ipsos shows exposure to misinformation has increased four fold between 2024-25. Misinformation on medicines and vaccines is rife, ranging from vaccines to paracetamol.

PIF’s response to the Medicines Health Care Regulatory Authority’s consultation on AI recommended models providing health advice should be regulated as medical devices. As the platforms start to offer new tools to make ‘sense of your health data’, the need for regulation becomes more pressing.

This would allow for safer use of AI, because the technology presents clear benefits.

It's not too late to act

In September 2024, PIF added requirements on the safe use of AI to the criteria for the PIF TICK, the UK trust mark for health information. We also published a framework for policy creation on the safe use of AI in health information. ⁹

Our latest data ¹⁰ suggests two in 10 organizations are using AI in some form to support creation of health information. Although people trust AI less than regulated sources, they love the ease of use. Credible health information producers must embrace the change.

At PIF, we are testing an AI agent with five PIF TICK-certified organizations. The agent can only access PIF certified content. It quickly provides accessible answers to queries posed by user testers. However, there are issues with tone of voice and sensitivity. Answer quality degrades as the agent reaches the limit of its knowledge base. This is the point when open models start to hallucinate.

We are now moving to phase two to resolve these issues. A model operating in Europe, Lupus GPT, also works on a closed content library and has answered 69,000 questions from people in 129 countries, expanding the reach of its trusted content.

In short, the benefits of AI are there to be harnessed but the models have to operate within a regulatory framework which supports quality, safety and health outcomes. The world failed to prevent harm from social media. It’s not too late to prevent this from happening with AI.

Sophie Randall will be joined at the upcoming event by The Times' Tom Whipple and two other experts. To find out more and confirm your attendance, click here. 

References

1 PIF IPSOS Knowledge is Power https://pifonline.org.uk/resources/knowledge-is-power/

2 PIF data to be published

3 Pew Centre https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-

likely-to-click-on-links-when-an-ai-summary-appears-in-the-results

4 BBC https://www.bbc.co.uk/news/articles/c8drzv37z4jo

5 PIF, Macmillan and Marie Curie, AI summaries https://pifonline.org.uk/resources/google-ai-

summaries/

6 The Guardian, https://www.theguardian.com/technology/2026/jan/11/google-ai-overviews-

health-guardian-investigation

7 Healthwatch https://www.healthwatchhalton.co.uk/news/2025-11-25/healthwatch-research-

shows-one-five-men-use-online-platforms-health-advice

8 Eve Appeal You Gov April 2026 (publication 30April)

9 Mind https://www.mind.org.uk/news-campaigns/news/mind-launches-ai-and-mental-health-

commission/#:~:text=Mind%20has%20announced%20that%20it,technological%20shifts%20of%

20our%20time.

10 PIF Framework on AI https://pifonline.org.uk/resources/balancing-the-risks-and-benefits-of-

ai-in-the-production-of-health-information-1/

11 PIF Health and Digital Literacy Survey 2026 https://pifonline.org.uk/resources/health-and-

digital-literacy-survey-202526/