To address the limitations of strong feature dependency,insufficient generalization,and inadequate perfor-mance in few-shot/cold-start settings in existing multimodal social media popularity prediction algorithms,a Mul-tiSmpLLM model based on large language model with instruction fine-tuning and human alignment was proposed.Firstly,the task of multimodal social media popularity prediction for cold-start users was defined.Secondly,multimodal fine-tuning instructions were constructed,and the large language model(Llama3)was instructionally fine-tuned using the low-rank adaptation(LoRA)and parameter freeze(Freeze)method.Finally,an improved direct preference optimiza-tion(DPO)algorithm IDPOP was developed by constructing preference data and adding a parameter-tuned penalty to the DPO loss function,resolving instability and non-convergence in RLHF and incorrect optimization in standard DPO for social media popularity prediction.Experiments show MultiSmpLLM outperforms conventional multimodal prediction models and multimodal large language models such as GPT-4o.
关键词
大语言模型/流行度预测/指令微调/人类对齐
Key words
large language model/popularity prediction/instruction fine-tuning/human alignment
维普
万方数据