首页|Depressive semantic awareness from vlog facial and vocal streams via spatio-temporal transformer

Depressive semantic awareness from vlog facial and vocal streams via spatio-temporal transformer

扫码查看
原文链接
  • 万方数据
  • 维普
With the rapid growth of information transmission via the Internet,efforts have been made to reduce network load to promote efficiency.One such application is semantic computing,which can extract and process semantic communication.Social media has enabled users to share their current emotions,opinions,and life events through their mobile devices.Notably,people suffering from mental health problems are more willing to share their feelings on social networks.Therefore,it is necessary to extract semantic information from social media(vlog data)to identify abnormal emotional states to facilitate early identification and intervention.Most studies do not consider spatio-temporal information when fusing multimodal information to identify abnormal emotional states such as depression.To solve this problem,this paper proposes a spatio-temporal squeeze transformer method for the extraction of semantic features of depression.First,a module with spatio-temporal data is embedded into the transformer encoder,which is utilized to obtain a representation of spatio-temporal features.Second,a classifier with a voting mechanism is designed to encourage the model to classify depression and non-depression effec-tively.Experiments are conducted on the D-Vlog dataset.The results show that the method is effective,and the accuracy rate can reach 70.70％.This work provides scaffolding for future work in the detection of affect recognition in semantic communication based on social media vlog data.

Emotional computingSemantic awarenessDepression recognitionVlog data

Yongfeng Tao、Minqiang Yang、Yushan Wu、Kevin Lee、Adrienne Kline、Bin Hu

展开 >

School of Information Science and Engineering,Lanzhou University,Lanzhou,China

The School of Accounting,Auditing and Taxation,Business School,UNSW Sydney,Australia

Department of Preventive Medicine,Northwestern University,Chicago,IL,United States

STI 2030-Major ProjectsNational Natural Science Foundation of ChinaNatural Science Foundation of Gansu Province,ChinaFundamental Research Funds for the Central UniversitiesSupercomputing Center of Lanzhou University

2021ZD02020026222780722JR5RA488lzujbky-2023-16

2024

10.1016/j.dcan.2023.03.007

数字通信与网络(英文)

数字通信与网络(英文)

ISSN：
年,卷(期)：2024.10(3)