大语言模型驱动的开源情报认知
Large language model-driven open-source intelligence cognition
张华平 1李春锦 1魏顺平 2耿国桐 3李伟伟 4李玉岗1
作者信息
- 1. 北京理工大学,北京 100081
- 2. 中央民族大学,北京 100081
- 3. 军事科学院军事科学信息研究中心,北京 100011
- 4. 军事科学院国防科技创新研究院,北京 100071
- 折叠
摘要
随着开源情报在军事领域的广泛应用,对相关情报的认知和分析需求日益增长.然而,当前研究人员所使用的大语言模型存在严重的幻觉现象,导致其生成的信息不可靠,无法直接用于军事开源情报的认知任务.为了解决这一问题,通过网上收集构建一个包含约10万条对话记录的军事开源情报数据集;利用LLaMA-13B模型作为基座,通过微调训练得到一个新的模型——ChatBIT,专门针对军事领域的对话和问答任务进行优化.对比分析ChatBIT模型与Vicuna-13B模型在军事知识问答方面的能力,通过一系列标准化的指标评估,包括Bleu值、Rouge-1、Rouge-2和Rouge-L,可知ChatBIT在所有指标上均优于Vicuna-13B.具体来说,相比Vicuna-13B,ChatBIT的Bleu值高2.3909,Rouge-1值高3.2079,Rouge-2值高2.2562,Rouge-L值高1.5939.结果表明,ChatBIT模型在处理军事领域的对话和问答任务时,能够提供更准确、更可靠的信息.
Abstract
With the extensive application of open-source intelligence in the military field,the demand for cognition and analysis of relevant intelligence is growing.However,the large language models currently used by researchers are prone to severe hallucination,rendering the information generated unreliable and unsuitable to direct utilization for the cognition of open-source military intelligence.To address this problem,the present study collected approximately 100,000 dialogue records online and constructed an open-source military intelligence dataset.Subsequently,a new model,ChatBIT,which is specifically optimized for dialogue and question answering tasks in the military field,was obtained by fine-tuning and training the LLaMA-13B base question answering model.This study further compared the military knowledge question answering capabilities of the ChatBIT model with those of the Vicuna-13B model.ChatBIT was found to outperform Vicuna-13B in a series of standardized evaluation metrics including the BLEU score,ROUGE-1,ROUGE-2,and ROUGE-L.Specifically,ChatBIT's BLEU score was 2.3909 higher than that of Vicuna-13B.Furthermore,ChatBIT's ROUGE-1,ROUGE-2,and ROUGE-L scores were respectively 3.2079,2.2562,and 1.5939 points higher than those of Vicuna-13B.These results indicate that the ChatBIT model provides more accurate and reliable information when dealing with military dialogue and question answering tasks.
关键词
大语言模型/ChatBIT/开源情报/人工智能Key words
large language model/ChatBIT/open-source intelligence/artificial intelligence引用本文复制引用
出版年
2024
NETL
NSTL
万方数据