March28, 2024

Assessing ChatGPT's Role in Medical Literature: Summarization Quality and Relevance Challenges

In a recent study published in The Annals of Family Medicine, researchers examined the effectiveness of Chat Generative Pretrained Transformer (ChatGPT) in summarizing medical abstracts to assist physicians in accessing concise, accurate, and unbiased summaries amidst the rapid expansion of clinical knowledge and limited review time.

Background:
The exponential growth of medical knowledge, coupled with clinical models prioritizing productivity, poses challenges for physicians to keep up with the literature. Artificial Intelligence (AI) tools like ChatGPT offer potential solutions. However, concerns exist regarding AI's ability to produce misleading or biased text.


Study Methodology:
Researchers selected 10 articles from each of 14 journals, aiming for diversity in topics and structures. They ensured ChatGPT had no prior exposure to the selected 2022 articles. ChatGPT summarized the articles, self-assessing quality, accuracy, and bias. Physician reviewers independently evaluated the summaries for quality, accuracy, bias, and relevance. Statistical and qualitative analyses compared ChatGPT's performance with human assessments.


Study Findings:
ChatGPT effectively condensed 140 medical abstracts from diverse journals, reducing them by 70%. Physicians rated the summaries highly for quality and accuracy, with minimal bias. Despite high ratings, some inaccuracies and hallucinations were identified, particularly in critical data omission and misinterpretation of study designs. ChatGPT's ability to recognize article relevance at the journal level aligned well with physician assessments. However, its performance in determining relevance to specific medical specialties was modest.