Beyond Unimodality:Advances and Prospects in Multimodal Computational Communication Research
With the rapid evolution of computational methods,there is a noticeable surge in Computational Communication Research(CCR)that transcends traditional text-based communication by delving into the visual aspects.However,the current focus remains predominantly unimodal,merely transitioning from text to images.Confronted with the inherently multimodal nature of media consumption,this article underscores the imperative of prioritizing research in multimodal communication.This study presents a comprehensive review of 31 multimodal computational communication research papers published in SSCI and A&HCI journals.The key findings are as follows:(1)Multimodal computational communication research is a relatively nascent field that demands further development;(2)Research patterns can be categorized into four groups based on the relationship between modalities(independent/combinational)and research objectives(descriptive/explanatory);(3)The primary source of research data is image-text-based social media;and(4)Variables generated through the combination of multimodal features include image-text congruence,image-text themes,image-text narratives,and aggressiveness.The article concludes with actionable suggestions,urging the application of generative AI,the advancement of theory construction,and the promotion of cross-platform and cross-regional development in the realm of multimodal computational communication research.