The Object tracking is one of the key tasks in building complex motion analysis systems in computer vision.Sports competitions are deeply loved by audiences because of their fun.Tracking sports objects in sports videos has broad application prospects in post-match review,technology improvement,sports education and media communication.Significant progress has been made in sports video tracking based on deep learning in recent years.This paper first introduced the research background of object tracking in sports videos,and gave the definition and classification of objects tracking based on deep learning.Secondly,we summarized the research status of object tracking in sports video from three aspects:ball tracking,single-camera player tracking and multi-camera multi-player tracking,extracted a unified algorithm process for every task,and conducted technical analysis and summary.Finally,we assessed the existing challenges and forecast the future directions of visual object tracking in sports videos.