Research on Chinese Medical Named Entity Recognition with Fusion of Multimodal Data
[Purpose/significance]Medical named entity recognition is a critical step for medical and healthcare knowledge mining and knowledge organization.The semantic associations between multimodal data are mined to improve the medical entity rec-ognition effect,which provides support for domain knowledge complementation and knowledge reasoning.[Method/process]In this paper,we propose a multimodal Chinese medical named entity recognition model BAF-MNER(Bilinear Attention Fusion-Multi-modal Named Entity Recognition)based on bilinear attention fusion mechanism.The model first learns semantic features from multi-modal medical data through visual and text encoders;next a bilinear attention network is utilized to achieve cross-modal semantic interaction between image and text,introducing a gating mechanism to filter the visual noise;then fusing the visual and text fea-tures based on the attention mechanism to construct the multimodal feature representations,and adding a batch normalization layer to optimize the deep neural network;finally,inputting the multimodal feature vector into a CRF layer to decode to obtain the pre-dicted labels.[Result/conclusion]The proposed model can effectively improve the Chinese medical named entity recognition,in-creasing the F1 value on multimodal medical dataset by 4.07％compared with the unimodal baseline model,and 1.65％compared with the multimodal baseline model;the experiments on multimodal public dataset indicate an excellent generalization of our model.
multimodal named entity recognitionmultimodal learningmultimodal fusionresidual networkbilinear atten-tion mechanism
万方数据
维普
