首页|引入反事实基线的无人机集群对抗博弈方法

引入反事实基线的无人机集群对抗博弈方法

扫码查看
原文链接
  • NETL
  • NSTL
  • 万方数据
无人机在协同对抗博弈上的应用越来越广泛和深入，尤其是无人机集群在协同探测、全域对抗、策略骗扰等对抗任务中，发挥着越来越重要作用，可靠高效的无人机集群博弈方法是当前的研究热点。本文将反事实基线思想引入到无人机集群对抗博弈环境，提出一种基于反事实多智能体策略梯度(counterfactual multi-agent policy gradients，COMA)的无人机集群对抗博弈方法;在具有无限连续状态、动作的无人机对抗环境中，基于无人机动力学模型，设置符合实际环境的击敌条件和奖励函数，构建基于多智能体深度强化学习的无人机集群对抗博弈模型。红蓝双方无人机集群采取不同的对抗博弈方法，利用多智能体粒子群环境(multi-agent particle environment，MPE)对红蓝双方无人机集群进行非对称性对抗实验，实验结果表明平均累积奖励能够收敛到纳什均衡，在解决4 vs。8的对抗决策问题方面，COMA方法的平均命中率较DQN和MADDPG分别提升39％和17％，在平均胜率方面比DQN和MADDPG分别提升34％和17％。最后，通过对COMA方法的收敛性和稳定性的深入分析，保证了 COMA方法在无人机集群对抗博弈任务上的实用性和鲁棒性。
UAV swarm adversarial game method with a counterfactual baseline
The collaborative adversarial game of unmanned aerial vehicles(UAVs)is becoming increasingly widespread and profound,especially in collaborative detection,global confrontation,strategic deception and other confrontation tasks.Reliable and efficient UAV swarm game methods are currently a hot research topic.This paper introduces the counterfactual baseline concept into the UAV swarm adversarial environment and proposes a UAV swarm adversarial game method based on counterfactual multi-agent policy gradients(COMA).In the UAV confrontation environment with infinite continuous states and actions,merging the UAV dynamics,we set up realistic attack conditions and reward functions,and construct a UAV swarm adversarial game model based on multi-agent deep reinforcement learning.The red and blue UAVs adopt different adversarial game methods,and asymmetric adversarial experiments are conducted in the multi-agent particle environment(MPE).The experimental results show that the average cumulative rewards can converge to Nash equilibrium.For 4 vs.8 adversarial decision-making scene,the average hit rate of COMA is 39％and 17％higher than that of DQN and MADDPG,while the average win rate is 34％and 17％higher than that of DQN and MADDPG,respectively.Finally,the practicality and robustness for UAV swarm adversarial game tasks are ensured through an in-depth analysis of the convergence and stability of COMA.

UAV swarmconfrontation gamemulti-agentdeep reinforcement learningNash equilibrium

王尔申、陈纪浩、宏晨、刘帆、陈艾东、景竑元

展开 >

沈阳航空航天大学电子信息工程学院,沈阳 110136

沈阳航空航天大学民用航空学院,沈阳 110136

北京联合大学多智能体系统研究中心,北京 100101

北京联合大学机器人学院,北京 100101

展开 >

无人机集群 对抗博弈 多智能体 深度强化学习 纳什均衡

国家重点研发计划国家自然科学基金北京联合大学科研北京联合大学科研北京联合大学科研北京联合大学科研卫星导航系统与装备技术国家重点实验室开放基金项目辽宁省属本科高校基本科研业务费专项辽宁省属本科高校基本科研业务费专项沈阳市科技计划项目

2018AAA010080462173237ZK30202304SK160202103ZK50201911ZK30202107CEPNT2022A01202401772024021522-322-3-34

2024

10.1360/SSI-2023-0305

中国科学F辑
中国科学院,国家自然科学基金委员会

中国科学F辑

CSTPCD北大核心
影响因子：1.438
ISSN：1674-5973
年,卷(期)：2024.54(7)