A trajectory optimization method for relay unmanned aerial vehicle(UAV)is proposed to overcome the problem of co-channel interference deteriorating link transmission reliability of UAV relay communication system.First,the point-to-point model of UAV relay communication system is established,and based on the criterion of maximizing the input signal-to-interference-plus-noise ratio(SINR)of the base station node demodulator,a trajectory opti-mization method for relay UAV is proposed.The optimal flying trajectory of the relay UAV is obtained using the particle swarm optimization(PSO)algorithm,and the link outage probability and ergodic capacity of the UAV relay communication system are given via theoretical analysis.Finally,the correctness of the proposed trajectory opti-mization method is verified through simulation.The results show that the proposed method can overcome the effect of co-channel interference and improve the reliability of system link transmission reliability.