Research on multimodal social media information popularity prediction based on large language model
To address the limitations of strong feature dependency,insufficient generalization,and inadequate perfor-mance in few-shot/cold-start settings in existing multimodal social media popularity prediction algorithms,a Mul-tiSmpLLM model based on large language model with instruction fine-tuning and human alignment was proposed.Firstly,the task of multimodal social media popularity prediction for cold-start users was defined.Secondly,multimodal fine-tuning instructions were constructed,and the large language model(Llama3)was instructionally fine-tuned using the low-rank adaptation(LoRA)and parameter freeze(Freeze)method.Finally,an improved direct preference optimiza-tion(DPO)algorithm IDPOP was developed by constructing preference data and adding a parameter-tuned penalty to the DPO loss function,resolving instability and non-convergence in RLHF and incorrect optimization in standard DPO for social media popularity prediction.Experiments show MultiSmpLLM outperforms conventional multimodal prediction models and multimodal large language models such as GPT-4o.
large language modelpopularity predictioninstruction fine-tuninghuman alignment
万方数据
维普
