Comparative Study on ChatGPT and Scholars'Abstracts:Taking the Field of Information Resource Man-agement as an Example
[Purpose/Significance]To explore the similarities and differences of Chinese abstracts written by ChatGPT and scholars can provide references for AI-generated academic paper detection and related re-search.[Method/Process]Firstly,taking the field of information resource management as an example,this paper extracted 500 highly cited papers from library science,information science,and archival science in the recent years.Based on the obtained paper titles,it used the prompt method to apply the ChatGPT tool to generate cor-responding abstract texts and construct a dataset.Secondly,it adopted 9 machine learning and deep learning al-gorithms to classify and detect abstract texts generated by ChatGPT and written by scholars.Finally,it analyzed and revealed the similarities and differences between the two from multiple perspectives,including text features,topic models,and ROUGE evaluation.[Result/Conclusion]Mainstream machine learning and deep learning algorithms trained on datasets can effectively distinguish whether abstracts are generated by AI or written by scholars,with BERT and ERNIE performing best,while RF and Xgboost best in machine learning algorithms.The number of abstract characters and sentences generated by ChatGPT is more than that written by scholars,and the keywords are mostly template-based words.The themes of the two are mostly the same,but different in"disciplinary system"and"digital humanities".The quantitative analysis of ROUGE and cosine similarity indi-cates that the abstracts generated by ChatGPT have a significant"shape-like"rather than"spirit-like"to that by scholars.
NETL
NSTL
万方数据
