As artificial intelligence (AI) becomes increasingly integrated into healthcare, effective prompting has
emerged as a key factor in optimizing large language model (LLM) performance. While strategies such as
chain-of-thought prompting can enhance reasoning, little is known about how medical students naturally
construct prompts when engaging with LLMs. Given ChatGPT’s ability to pass the USMLE, the research
question guiding this study was “How do medical students interact with LLMs, and how does prior digital
health training influence these approaches?” To address this, we examined how medical students at
Rocky Vista University used ChatGPT-4.0 to answer medically related questions. The primary objective
was to characterize prompt themes and elements; the secondary objective was to compare students
enrolled in a longitudinal digital health curriculum—including training in prompt engineering—with
peers lacking formal instruction. An 11-question Qualtrics survey was distributed across the Colorado and
Utah campuses, including six demographic items and five medical questions. Of 108 responses, 60 met
eligibility criteria and were analyzed. Responses were evaluated for prompting styles, AI interactions, and
digital health participation. Findings revealed challenges for both students and ChatGPT in osteopathic
principles and practice (OPP), particularly with sacral landmarks and axis application (correct response
rate ~52%). In contrast, ChatGPT answered an ethics-based question correctly that many students
misinterpreted, highlighting differences in reasoning rather than model performance. Prompting strategy
influenced outcomes: students using targeted prompts with copy-and-paste achieved the highest accuracy
(75% fully correct), while most relied on unmodified copy-and-paste. Limitations include the small
sample size and recruitment from a single medical program, which may limit generalizability. This study
suggests that targeted prompt design improves LLM accuracy and that incorporating structured prompting
instruction may be critical to preparing medical students to engage with AI responsibly and productively
in medical education.
