Importance: Timely access to human expertise for affordable and efficient triage of ophthalmic conditions is inconsistent. With recent advancements in publicly available artificial intelligence (AI) chatbots, individuals may turn to these tools for triage of ophthalmic complaints. Validation studies are necessary to evaluate the performance of AI chatbots as triage tools and inform the public regarding their safety.
Objective: To evaluate the triage performance of AI chatbots for ophthalmic conditions
Design: Cross-sectional study
Setting: Single center
Participants: Ophthalmology trainees, OpenAI ChatGPT (GPT-4), Bing Chat, and WebMD Symptom Checker
Methods: 44 clinical vignettes representing common ophthalmic complaints were developed, and a standardized pathway of prompts were presented to each tool in March 2023. Primary outcomes were proportion of responses with correct diagnosis listed in the top three possible diagnoses, and proportion with correct triage urgency. Ancillary outcomes included presence of grossly inaccurate statements, mean reading grade level, mean response word count, proportion with attribution, most common sources cited, and proportion with a disclaimer regarding chatbot limitations in dispensing medical advice.
Results: The ophthalmologists-in-training, ChatGPT, Bing Chat, and WebMD listed the appropriate diagnosis among the top three suggestions in 42 (95%), 41 (93%), 34 (77%), and 8 (33%) cases, respectively. Triage urgency was appropriate in 38 (86%), 43 (98%), and 37 (84%) cases for the ophthalmology trainees, ChatGPT, and Bing Chat, correspondingly.
Conclusions: ChatGPT using the GPT-4 model offered high diagnostic and triage accuracy that was comparable to ophthalmology trainees, with no grossly inaccurate statements. Bing Chat had lower accuracy and a tendency to overestimate triage urgency.
Canadian Journal of Ophthalmology (CJO) / 2023