首页|Multimodal Distillation Pre-Training Model for Ultrasound Dynamic Images Annotation

Multimodal Distillation Pre-Training Model for Ultrasound Dynamic Images Annotation

扫码查看
原文链接
  • NETL
  • NSTL
  • IEEE
With the development of medical technology, ultrasonography has become an important diagnostic method in doctors' clinical work. However, compared with the static medical image processing work such as CT, MRI, etc., which has more research bases, ultrasonography is a dynamic medical image similar to video, which is captured and generated by a real-time moving probe, so how to deal with the video data in the medical field and cross modal extraction of the textual semantics in the medical video is a difficult problem that needs to be researched. For this reason, this paper proposes a pre-training model of multimodal distillation and fusion coding for processing the semantic relationship between ultrasound dynamic Images and text. Firstly, by designing the fusion encoder, the visual geometric features of tissues and organs in ultrasound dynamic images, the overall visual appearance descriptive features and the named entity linguistic features are fused to form a unified visual-linguistic feature, so that the model obtains richer visual, linguistic cues aggregation and alignment ability. Then, the pre-training model is augmented by multimodal knowledge distillation to improve the learning ability of the model. The final experimental results on multiple datasets show that the multimodal distillation pre-training model generally improves the fusion ability of various types of features in ultrasound dynamic images, and realizes the automated and accurate annotation of ultrasound dynamic images.

VisualizationBiomedical imagingComputational modelingTrainingMedical diagnostic imagingSemanticsAnnotations

Xiaojun Chen、Jia Ke、Yaning Zhang、Jianping Gou、Anna Shen、Shaohua Wan

展开 >

Affiliated Hospital of Jiangsu University, Zhenjiang, China|School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang, China|School of Management, Jiangsu University, Zhenjiang, China

School of Management, Jiangsu University, Zhenjiang, China

College of Computer and Information Science, College of Software, Southwest University, Chongqing, China

Affiliated Hospital of Jiangsu University, Zhenjiang, China

Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen, China

展开 >

2025

10.1109/JBHI.2024.3438254

IEEE journal of biomedical and health informatics

IEEE journal of biomedical and health informatics

ISSN：
年,卷(期)：2025.29(5)
  • 51