双注意力记忆多智能体强化学习
BiTransformer Memory for Multi-agent Reinforcement Learning
马裕博 1周长东 1张志文 1杨培泽 1张博1
作者信息
- 1. 大连海事大学人工智能学院,大连 116026
- 折叠
摘要
多智能体协同在强化学习研究领域占据重要地位,旨在深入探讨智能体如何通过相互协作实现共同目标.大部分协作多智能体算法注重合作的构建,但忽略了个体策略的强化.为解决上述问题,本文提出一种BiTransformer记忆(BTM)在线强化学习模型,该模型不仅考虑多智能体之间的协同,还利用记忆模块辅助个体决策.BTM由双注意力编码器和双注意力解码器组成,分别用于个体策略的增强和多智能体系统的协作.在双注意力编码器中,受人类的决策经验依赖的启发,提出记忆注意力模块为当前决策提供历史决策经验.与传统利用RNN的方法不同,BTM为每一个提供的是一个显式历史决策经验库,而非隐藏单元.此外,提出融合注意力模块,在历史决策经验的辅助下处理当下的局部观测信息,从而获取环境中最具决策价值的信息,进一步提高智能体个体的决策能力.在双注意力解码器中,本文提出了决策注意力模块和合作注意力模块两个模块,通过综合考虑其他已经做出决策智能体与当前智能体的合作收益以及带有历史决策经验的局部观察,从而促进历史决策辅助下的多智能体潜在合作的形成.最终本文在星际争霸中的多个场景下对BTM进行了测试,取得了93%的平均胜率.
Abstract
Multi-agent collaboration plays a crucial role in the field of reinforcement learning,focusing on how agents cooperate to achieve common goals.Most collaborative multi-agent algorithms emphasize the construction of collaboration but overlook the reinforcement of individual decision-making.To address this issue,this study proposes an online reinforcement learning model,BiTransformer memory(BTM),which not only considers the collaboration among multiple agents but also uses a memory module to assist individual decision-making.The BTM model is composed of a BiTransformer encoder and a BiTransformer decoder,which are utilized to improve individual decision-making and collaboration within the multi-agent system,respectively.Inspired by human reliance on historical decision-making experience,the BiTransformer encoder introduces a memory attention module to aid current decisions with a library of explicit historical decision-making experience rather than hidden units,differing from the conventional RNN-based method.Additionally,an attention fusion module is proposed to process partial observations with the assistance of historical decision experience,to obtain the most valuable information for decision-making from the environment,thereby enhancing the decision-making capabilities of individual agents.In the BiTransformer decoder,two modules are proposed:a decision attention module and a collaborative attention module.They are used to foster potential cooperation among agents by considering the collaborative benefits between other decision-making agents and the current agent,as well as partial observations with historical decision-making experience.BTM is tested in multiple scenes of StarCraft,achieving an average win rate of 93%.
关键词
多智能体协同/在线强化学习/局部观测/历史决策经验/合作收益/个体策略增强Key words
multi-agent collaboration/online reinforcement learning/partial observation/historical decision-making experience/collaborative benefit/individual policy enhancement引用本文复制引用
出版年
2024
NETL
NSTL
万方数据