首页|Performance Assessment of GPT 4.0 on the Japanese Medical Licensing Examination

Performance Assessment of GPT 4.0 on the Japanese Medical Licensing Examination

扫码查看
原文链接
  • NETL
  • NSTL
  • 万方数据
Objective:To evaluate the accuracy and parsing ability of GPT 4.0 for Japanese medical practitioner qualification examinations in a multidimensional way to investigate its response accuracy and comprehensiveness to medical knowledge.Methods:We evaluated the performance of the GPT 4.0 on Japanese Medical Licensing Examination(JMLE)questions(2021-2023).Questions are categorized by difficulty and type,with distinctions between general and clinical parts,as well as between single-choice(MCQ1)and multiple-choice(MCQ2)questions.Difficulty levels were determined on the basis of correct rates provided by the JMLE Preparatory School.The accuracy and quality of the GPT 4.0 responses were analyzed via an improved Global Qualily Scale(GQS)scores,considering both the chosen options and the accompanying analysis.Descriptive statistics and Pearson Chi-square tests were used to examine performance across exam years,question difficulty,type,and choice.GPT 4.0 ability was evaluated via the GQS,with comparisons made via the Mann-Whitney U or Kruskal-Wallis test.Results:The correct response rate and parsing ability of the GPT4.0 to the JMLE questions reached the qualification level(80.4％).In terms of the accuracy of the GPT4.0 response to the JMLE,we found significant differences in accuracy across both difficulty levels and option types.According to the GQS scores for the GPT 4.0 responses to all the JMLE questions,the performance of the questionnaire varied according to year and choice type.Conclusion:GTP4.0 performs well in providing basic support in medical education and medical research,but it also needs to input a large amount of medical-related data to train its model and improve the accuracy of its medical knowledge output.Further integration of ChatGPT with the medical field could open new opportunities for medicine.

large language modelChatGPTartificial intelligenceJapanese medical licensing examination

Hong-lin WANG、Hong ZHOU、Jia-yao ZHANG、Yi XIE、Jia-ming YANG、Ming-di XUE、Zi-neng YAN、Wen LI、Xi-bao ZHANG、Yong WU、Xiao-ling CHEN、Peng-ran LIU、Lin LU、Zhe-wei YE

展开 >

Department of Orthopedics Surgery,Union Hospital,Tongji Medical College,Huazhong University of Science and Technology,Wuhan 430022,China

Laboratory of Intelligent Medicine,Union Hospital,Tongji Medical College,Huazhong University of Science and Technology,Wuhan 430022,China

Homes for the Elderly in Hainan Province,Hainan 570216,China

Hainan Wuhua Intelligent Design and Numerical Control Technology Research Institute,Hainan 571137,China

Department of Orthopedics,Renmin Hospital of Wuhan University,Wuhan 430060,China

展开 >

2024

10.1007/s11596-024-2932-9

当代医学科学(英文)
华中科技大学同济医学院

当代医学科学(英文)

影响因子：0.748
ISSN：2096-5230
年,卷(期)：2024.44(6)