In order to improve the safety supervision of crew members'activities during ship operation,the visual recognition technology for crew behavior recognition is developed.A YOLO algorithm improved in handling class-imbalance is introduced into the supervision system design.The introduction of the algorithm ensures the balance between the recognition accuracy and the computing time.The algorithm has been used in a chemical ship's safety supervision system for verification.The automatic recognition of human activity and early warning of abnormal situation have greatly enhanced the confidence in ship operation safety.