Browsing by Subject "Script concordance test"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Publication
    Open Access
    A Prompt for Generating Script Concordance Test UsingChatGPT, Claude, and Llama Large Language ModelChatbots
    (Universidad de Murcia. Servicio de publicaciones, 2024) Kıyak, Yavuz Selim; Emekli, Emre
    Medical education always evolves to incorporate more tools for specific needs in assessing clinicalreasoning skills. Among these tools, Script Concordance Test (SCT) has a particular importancedue to its focus on assessing decision-making in uncertain clinical situations. However,development of SCT items is effortful. Artificial intelligence tools, such as large language models,offer significant benefits. These models are already used for generating multiple-choice questions,and their use in generating SCTs offers great promise. However, this requires well-designedprompts to generate SCTs. This article proposes a generic prompt for the ChatGPT-4, Claude 3,Llama 3, and ChatGPT-4o large language model chatbots to generate SCTs, which can be tailoredto various fields of medicine and different stages of medical education. It can help to streamlinethe development process of SCTs. Initial findings are promising, and there is a need for generatingSCTs using large language models and conducting research to assess the quality of SCTs
  • Thumbnail Image
    Publication
    Open Access
    Using Large Language Models to Generate Script Concordance Test in Medical Education: ChatGPT and Claude
    (Universidad de Murcia. Servicio de publicaciones, 2025) Kıyak, Yavuz Selim; Emekli, Emre; Departamentos
    Using Large Language Models to Generate Script Concordance Test in Medical Education: ChatGPT and ClaudeYavuz Selim Kıyak1 *, Emre Emekli21Department of Medical Education and Informatics, Gazi University Faculty of Medicine, Ankara, Turkiye; yskiyak@gazi.edu.tr, 0000-0002-5026-32342Department of Radiology, Faculty of Medicine, Eskişehir Osmangazi University, Eskişehir, Turkiye; 0000-0001-5989-1897* Correspondence: yskiyak@gazi.edu.trReceived: 4/11/24; Accepted: 2/12/24; Published: 3/12/24AbstractWe aimed to determine the quality of AI-generated (ChatGPT-4 and Claude 3) Script ConcordanceTest (SCT) items through an expert panel.We generated SCT items on abdominal radiology usinga complex prompt in large language model (LLM) chatbots (ChatGPT-4 and Claude 3 (Sonnet) inApril 2024) and evaluated the items’ quality through an expert panel of 16 radiologists. Expertpanel, which was blind to the origin of the items provided without modifications, independentlyanswered each item and assessed them using 12 quality indicators. Data analysis includeddescriptive statistics, bar charts to compare responses against accepted forms, and a heatmap toshow performance in terms of the quality indicators. SCT items generated by chatbots assessclinical reasoning rather than only factual recall (ChatGPT: 92.50%, Claude: 85.00%). The heatmapindicated that the items were generally acceptable, with most responses favorable across qualityindicators (ChatGPT: 71.77%, Claude: 64.23%). The comparison of the bar charts with acceptableand unacceptable forms revealed that 73.33% and 53.33% of the questions in the items can beconsidered acceptable, respectively, for ChatGPT and Claude. The use of LLMs to generate SCTitems can be helpful for medical educators by reducing the required time and effort. Although theprompt provides a good starting point, it remains crucial to review and revise AI-generated SCTitems before educational use. The prompt and the custom GPT, “Script Concordance TestGenerator”, available at https://chatgpt.com/g/g-RlzW5xdc1-script-concordance-test-generator,can streamline SCT item development.