Artificial intelligence program passes exam to obtain medical license in the US
Time to Read: 1 minute
The ChatGPT chatbot scored between 52.4% and 75.0% in all three USMLE exams, when the pass threshold each year is about 60%
The use of artificial intelligence for the generation of images and writing has gained popularity in recent months thanks to the presence of programs with this technology, one of them is the ChatGPT chatbot, which recently achieved a passing grade for the Licensing Exam. United States Medical Association (USMLE).
According to a study published February 9, 2023, in the open access journal PLOS Digital Health by Tiffany Kung, Victor Tseng, and colleagues at AnsibleHealth, ChatGPT can score at or around the approximately 60 percent pass threshold for the USMLE, with answers that make coherent and internal sense and contain frequent ideas.
Kung and his colleagues tested ChatGPT's performance on the USMLE, a series of three highly standardized and regulated exams required for medical licensure in the United States.
Carried out by medical students and physicians-in-training, the USMLE tests knowledge spanning most medical disciplines, from biochemistry to diagnostic reasoning to bioethics.
After removing the image-based questions, the authors tested the software with 350 of the 376 public questions available in the June 2022 version of the USMLE.
Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelshttps://t.co/4ghHo4UmQV
— Wafik S. El-Deiry, MD, PhD, FACP, FRSM (@weldeiry) February 10, 2023
These were the results obtained by ChatGPT
After indeterminate responses were removed, ChatGPT scored between 52.4% and 75.0% across the three USMLE exams. The passing threshold each year is about 60%.
ChatGPT also demonstrated 94.6% agreement across all of their responses and produced at least meaningful knowledge (something that was new, non-obvious, and clinically valid) in 88.9% of their responses.
Notably, ChatGPT outperformed PubMedGPT, a counterpart model trained exclusively on biomedical literature, which scored 50.8% on an older dataset of USMLE-type questions.