基于三维人体姿态信息的协作式扩散试衣生成网络
A collaborative diffusion-based network for virtual try-on based on 3D human pose information
丁焕 1侯珏 2杨阳 2刘正3
作者信息
- 1. 浙江理工大学服装学院,杭州 311199
- 2. 浙江理工大学服装学院,杭州 311199;浙江理工大学丝绸文化传承与产品设计数字化技术文旅部重点实验室,杭州 311199
- 3. 浙江理工大学丝绸文化传承与产品设计数字化技术文旅部重点实验室,杭州 311199;浙江理工大学国际时装技术学院,杭州 311199
- 折叠
摘要
为解决人体关节遮挡产生的试衣错误、试衣生成结果保真度低以及试衣前后人体形态保持不一致等问题,提出了一种基于三维人体姿态信息的协作式扩散试衣生成网络-3DM-VTON网络.首先,通过三维人体姿态估计模型重构二维人体图像对应的三维人体姿态信息,获取人体各部位的空间顺序.其次,利用多部件服装变形网络,根据服装部件的外观流场对目标服装各部位进行变形,并在人体部位空间顺序的指导下对各服装部件特征进行融合.最后,将人体全局特征与融合后的服装特征引入扩散网络,通过人体姿态控制模块与区域度判别器限制其生成自由度,保证试衣前后人体形态一致性,从而提升图像局部细粒度,获得高质量的虚拟试衣结果.将3DM-VTON网络与目前常用的二维虚拟试衣网络进行实验对比,结果表明:3DM-VTON网络在人体关节遮挡处的服装变形准确度更高,试衣结果更加清晰、自然和逼真,试衣前后人体形态保持更一致.3DM-VTON网络能够生成高质、多姿势的试衣结果,满足虚拟试衣用户更多的试穿需求,可以提升消费者在线购物体验度,助推服装电商行业发展.
Abstract
To address the issues of clothing errors caused by joint occlusion,low fidelity of try-on results,and poor consistency in human body shape before and after try-on,a collaborative diffusion-based virtual try-on network using 3D human body pose information,termed 3DM-VTON,was proposed.Firstly,a 3D human body pose estimation network was used to reconstruct the 3D pose information corresponding to the 2D human images,and the spatial order of different body parts was acquired.Secondly,a multi-part garment deformation network was used to deform each part of the target garment based on local appearance flow fields and to integrate the features of each garment component under the guidance of the spatial relationship of body parts.Finally,the garment features integrating global human features with local garment features were introduced into the diffusion-based network,and the generation freedom of the diffusion network was constrained by a human body pose control network and a regional fidelity discriminator,ensuring consistency in human body shape before and after try-on,enhancing local granularity of the image,and achieving high-quality virtual try-on results.Comparative experiments with 3DM-VTON and the currently commonly used 2D virtual try-on network showed that 3DM-VTON achieved higher accuracy in garment deformation at joint occlusions,provided clearer,more natural,and realistic overall try-on results,and maintained better consistency in human body shape before and after try-on.The 3DM-VTON can generate high-quality,multi-posture try-on results to meet the needs of virtual try-on users,which can improve consumers'online shopping experience and boost the development of the clothing e-commerce industry.
关键词
虚拟试衣/生成对抗网络/扩散网络/外观流场/服装部件分割Key words
virtual try-on/generative adversarial network/diffusion network/appearance flow/garment part segmentation引用本文复制引用
出版年
2024
NETL
NSTL
万方数据