This article addresses the issue that traditional network monitoring technologies struggle to meet current security needs,and designs a computer network monitoring system that integrates artificial intelligence technology.Utilizing deep learning and reinforcement learning algorithms,the system can efficiently identify abnormal behaviors and intelligently respond to security incidents.Experimental results show that the Transformer-based model significantly outperforms traditional methods in detection accuracy and response efficiency.Moreover,the system has strong scalability and adaptability,which can enhance monitoring effects in complex network environments.