Multimodal recognition method based on heterogeneous feature deconstruction
In order to solve the problem of feature redundancy in the process of extracting heterogeneous features due to the difference between the distribution of heterogeneous modal data,this paper proposes a multi-modal recognition method based on the deconstruction of heterogeneous features.That is,build a heterogeneous feature deconstruction(HFD)model,train the feature extractor through gradient descent,and train the common feature extractor through gradient inversion to extract modal characteristic features with different modal characteristics.And modal common features with modal invariable properties,further use the common features to enhance the loss,improve the similarity between common features,and solve the problem of high redundancy between heterogeneous features.The results of comparison and ablation experiments on the CMU-MOSEI dataset verify that the proposed multi-modal recognition method based on heterogeneous feature deconstruction can effectively improve the recognition performance.
multimodal fusionheterogeneous featurefeature extractiongradient reversecosine similarityemotion recognitionfeature deconstructionmodal invariant space
万方数据
维普
