Research on Evaluation of Chinese Stylistic Competence on Large Language Models
Stylistic competence is an important pragmatic competence,and full evaluation and development of stylistic competence is required for Large Language Model(LLM)to function effectively in everyday language contexts.In this paper,stylistic competence is defined as the ability to use appropriate style for communication in a specific register.Based on this,three tasks of stylistic classification,stylistic generation and stylistic transformation are designed to evaluate the Chinese stylistic competence of LLMs represented by ChatGPT.It is found that LLMs have their own advantages and limitations in different tasks and styles.GPT-4 demonstrates the most comprehensive and excellent Chinese stylistic competence,while ChatGPT3.5 and ERNIE Bot have better performances.On the other hand,ChatGLM-6B and SparkDesk have weak and unstable performances.In addition,the prose,novel and other texts generated by each model are too formal.The literary grace is ordinary,and problems such as consistency errors,normative errors,factual errors,illogicality,insufficient sentence fluency and obvious traces of machine translation still exist.This study also provides methodological references for training and testing human stylistic competence,and carries reference value for language competence enhancement in the fields of Chinese teaching and international Chinese language education.
Large Language Modelstylistic competencelanguage resource
NETL
NSTL
万方数据
