Woodpecker:hallucination correction for multimodal large language models

扫码查看

原文链接

万方数据
维普

外文摘要：Hallucinations is a big shadow hanging over the rapidly evolving multimodal large language models(MLLMs),referring to that the generated text is inconsistent with the image content.To miti-gate hallucinations,existing studies mainly resort to an instruction-tuning manner that requires retraining the models with specific data.In this paper,we pave a different way,introducing a training-free method named Woodpecker.Like woodpeckers heal trees,it picks out and corrects hallucinations from the generated text.Concretely,Woodpecker consists of five stages:key concept extraction,question formulation,visual knowledge validation,visual claim generation,and hallucination correction.Implemented in a post-remedy manner,Woodpecker can easily serve different MLLMs,while being interpretable by accessing intermediate outputs of the five stages.We evaluate Woodpecker both quantitatively and qualitatively and show the huge potential of this new paradigm.On the POPE benchmark,our method obtains a 30.66％/24.33％improve-ment in accuracy over the baseline MiniGPT-4/mPLUG-Owl.The source code is released at https://github.com/BradyFU/Woodpecker.

外文关键词：

multimodal learningmultimodal large language modelshallucination correctionlarge lan-guage modelsvision and language

作者：

Shukang YIN、Chaoyou FU、Sirui ZHAO、Tong XU、Hao WANG、Dianbo SUI、Yunhang SHEN、Ke LI、Xing SUN、Enhong CHEN

展开 >

作者单位：

School of Artificial Intelligence and Data Science,University of Science and Technology of China,Hefei 230026,China

State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing 210023,China

School of Intelligence Science and Technology,Nanjing University,Suzhou 215163,China

Institute of Automation,Chinese Academy of Sciences,Beijing 100190,China

YouTu,Shanghai 200233,China

展开 >

出版年：

2024

DOI：

10.1007/s11432-024-4251-x

中国科学:信息科学(英文版)

中国科学院

中国科学:信息科学(英文版)

CSTPCDEI

影响因子：0.715

ISSN：1674-733X

年,卷(期)：2024.67(12)