首页|Cooperative decision-making algorithm with efficient convergence for UCAV formation in beyond-visual-range air combat based on multi-agent reinforcement learning

Cooperative decision-making algorithm with efficient convergence for UCAV formation in beyond-visual-range air combat based on multi-agent reinforcement learning

扫码查看
原文链接
  • NETL
  • NSTL
  • 万方数据
Highly intelligent Unmanned Combat Aerial Vehicle(UCAV)formation is expected to bring out strengths in Beyond-Visual-Range(BVR)air combat.Although Multi-Agent Reinforce-ment Learning(MARL)shows outstanding performance in cooperative decision-making,it is chal-lenging for existing MARL algorithms to quickly converge to an optimal strategy for UCAV formation in BVR air combat where confrontation is complicated and reward is extremely sparse and delayed.Aiming to solve this problem,this paper proposes an Advantage Highlight Multi-Agent Proximal Policy Optimization(AHMAPPO)algorithm.First,at every step,the AHMAPPO records the degree to which the best formation exceeds the average of formations in parallel envi-ronments and carries out additional advantage sampling according to it.Then,the sampling result is introduced into the updating process of the actor network to improve its optimization efficiency.Finally,the simulation results reveal that compared with some state-of-the-art MARL algorithms,the AHMAPPO can obtain a more excellent strategy utilizing fewer sample episodes in the UCAV formation BVR air combat simulation environment built in this paper,which can reflect the critical features of BVR air combat.The AHMAPPO can significantly increase the convergence efficiency of the strategy for UCAV formation in BVR air combat,with a maximum increase of 81.5％rel-ative to other algorithms.

Unmanned combat aerial vehicle(UCAV)formationDecision-makingBeyond-visual-range(BVR)air combatAdvantage highlightMulti-agent reinforcement learning(MARL)

Yaoming ZHOU、Fan YANG、Chaoyue ZHANG、Shida LI、Yongchao WANG

展开 >

School of Aeronautic Science and Engineering,Beihang University,Beijing 100191,China

Key Laboratory of Industrial Control Technology,Institute of Cyber-Systems and Control,Zhejiang University,Hangzhou 310027,China

National Natural Science Foundation of ChinaAeronautical Science Foundation of ChinaFundamental Research Funds for the Central Universities,China

5227238220200017051001

2024

10.1016/j.cja.2024.04.008

中国航空学报(英文版)
中国航空学会

中国航空学报(英文版)

CSTPCDEI
影响因子：0.847
ISSN：1000-9361
年,卷(期)：2024.37(8)
  • 1