首页|Synchronous Multi-Modal Semantic Communication System With Packet-Level Coding
Synchronous Multi-Modal Semantic Communication System With Packet-Level Coding
扫码查看
点击上方二维码区域，可以放大扫码查看
原文链接
NETL
NSTL
IEEE
Although the semantic communication with joint semantic-channel coding design has shown promising performance in transmitting data of different modalities over physical layer channels, the synchronization and packet-level forward error correction (FEC) of multimodal semantics have not been well studied. Synchronizing multimodal features in both the semantic and time domains is challenging due to the independent design of semantic encoders. In this paper, we take the facial video and speech transmission as an example and propose a Synchronous Multi-modal Semantic Communication System with Packet-Level Coding (SyncSC). To achieve semantic and time synchronization, 3D Morphable Mode (3DMM) coefficients and text are transmitted as semantics. We propose a semantic codec that achieves similar reconstruction quality with lower bandwidth. The visual-guided speech synthesis is designed to synchronize video, text and speech. We propose a packet-Level FEC method for video semantics, called PacSC, that maintains visual quality even at high packet loss rates. For text packets, a text packet loss concealment module, called TextPC, based on Bidirectional Encoder Representations from Transformers (BERT) is proposed, which improves the performance of traditional FEC methods. Simulation results show that SyncSC reduces transmission overhead while ensuring high-quality synchronous transmission of video and speech over the packet loss network.
Yun Tian、Jingkai Ying、Zhijin Qin、Ye Jin、Xiaoming Tao
展开 >
School of Electronics, Peking University, Beijing, China
Department of Electronic Engineering, Tsinghua University, Beijing, China|State Key Laboratory of Space Network and Communications, Beijing, China|Beijing National Research Center for Information Science and Technology, Beijing, China
NETL
NSTL
IEEE
