首页|Multimodal fusion recognition for digital twin

Multimodal fusion recognition for digital twin

扫码查看
原文链接
  • NETL
  • NSTL
  • 万方数据
The digital twin is the concept of transcending reality,which is the reverse feedback from the real physical space to the virtual digital space.People hold great prospects for this emerging technology.In order to realize the upgrading of the digital twin industrial chain,it is urgent to introduce more modalities,such as vision,haptics,hearing and smell,into the virtual digital space,which assists physical entities and virtual objects in creating a closer connection.Therefore,perceptual understanding and object recognition have become an urgent hot topic in the digital twin.Existing surface material classification schemes often achieve recognition through machine learning or deep learning in a single modality,ignoring the complementarity between multiple modalities.In order to overcome this dilemma,we propose a multimodal fusion network in our article that combines two modalities,visual and haptic,for surface material recognition.On the one hand,the network makes full use of the potential correlations between multiple modalities to deeply mine the modal semantics and complete the data mapping.On the other hand,the network is extensible and can be used as a universal architecture to include more modalities.Experiments show that the constructed multimodal fusion network can achieve 99.42％classification accuracy while reducing complexity.

Digital twinMultimodal fusionObject recognitionDeep learningTransfer learning

Tianzhe Zhou、Xuguang Zhang、Bing Kang、Mingkai Chen

展开 >

Key Laboratory of Broadband Wireless Communication and Sensor Network Technology,Ministry of Education,Nanjing University of Posts and Telecommunications,Nanjing 210003,China

National Natural Science Foundation of ChinaNational Natural Science Foundation of ChinaNational Natural Science Foundation of ChinaKey R & D Program of Jiangsu Province Key project and topicsNatural Science Foundation of Jiangsu Province Higher Education InstitutionsFuture Network Scientific Research Fund Projectopen research fund of Key Lab of Broadband Wireless Communication and Sensor Network TechnologyNUPTSF

620012466200124862171232BE202109520KJB510020FNSRFP-2021-YB-16JZNY202110NY220070

2024

10.1016/j.dcan.2022.10.009

数字通信与网络(英文)

数字通信与网络(英文)

ISSN：
年,卷(期)：2024.10(2)
  • 54