Performance and Challenges of InstructGPT in Named Entity Recognition
Currently,the research on Large Language Models(LLMs),such as InstructGPT,is primarily focused on free-form generation tasks,while the exploration in structured extraction tasks has been overlooked.In order to gain a deep understanding of LLMs on structured extraction tasks,this paper analyzes InstructGPT's performance on named entity recognition(NER),one of the fundamental structured extraction tasks,in both zero-shot and few-shot settings.To ensure the reliability of the findings,the experiments cover common and nested datasets from both biomedical domain and general domain.The results demonstrate that InstructGPT's performance on zero-shot NER achieves 11%to 56%of the performance by a finetuned small-scaled model.To explore why InstructGPT struggles with NER,this paper examines the outputs,finding invalid generation for 50%of them.Besides,the occurrence of both"false-negative"and"false-positive"predictions makes it difficult to improve performance by only addressing the invalid generation.Therefore,in addition to ensuring the validity of generated outputs,further research still should focus on finding effective ways of using InstructGPT in this area.
large language modelnamed entity recognitionin-context learningchain-of-thought
NETL
NSTL
万方数据
