Large Language Model-Driven Academic Text Mining:Construction and Evaluation of Inference-End Prompting Strategy
Task comprehension and instruction-following abilities of large language models enable users to complete complex information-processing tasks through simple interactive instructions.Scientific literature analysts are actively ex-ploring the application of large language models;however,a systematic study of the capability boundaries of large mod-els has not yet been conducted.Focusing on academic text mining,this study designs inference-end prompting strategies and establishes a comprehensive evaluation framework for large language model-driven academic text mining,encom-passing text classification,information extraction,text reasoning,and text generation,covering six tasks in total.Main-stream instruction-tuned models were selected for the experiments,to compare the different prompting strategies and pro-fessional capabilities of the models.The experiments indicate that complex instruction strategies,such as few-shot and chain-of-thought,are not effective in classification tasks,but perform well in more challenging tasks,such as extraction and generation,whereby trillion-parameter scale models achieve results comparable to those of fully trained deep-learn-ing models.However,for models with billions or tens of billions of parameter scales,there is a clear upper limit to infer-ence-end instruction strategies.Achieving deep integration of large language models into the field of scientific intelli-gence requires adaption of the model to the domain at the tuning end.
large language modelacademic text mininginstruction engineeringcapability evaluation
NETL
NSTL
万方数据
