首页|基于强化学习的离散事件系统最优定向监控

基于强化学习的离散事件系统最优定向监控

扫码查看
原文链接
  • 万方数据
  • 维普
对于多个可控事件(控制指令)允许同时执行的情形,离散事件系统的监控器进行随机选择.然而,在实际应用中,如交通调度、机器人路径规划,可控事件的定向选择和数值优化是必须要考虑和解决的两个问题.对此,引入一种优化机制量化控制成本,将监督控制理论与强化学习结合,提出一种基于强化学习的离散事件系统最优定向监控器求解方法,使被控系统实现以下三个目标:(1)遵循安全性和活性控制规范;(2)每个状态下至多允许一个可控事件执行;(3)从初始状态到标记状态事件执行累计成本最小.首先,建立系统和控制规范的自动机模型,做同步积运算后可得到目标模型,通过定义的成本函数为目标模型中每个事件的执行赋予成本.其次,利用监督控制理论求解无阻塞且行为最大许可的监控器.最后,将监控器转化为马尔可夫决策过程,并利用Q学习算法求解出最优定向监控器.使用单向列车导轨控制案例和多轨道列车控制案例验证所提方法的有效性和正确性.仿真结果表明,所提出方法能够实现系统的无阻塞定向控制,并且使得定向监控器的数值成本最小.
Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning
In the case that several controllable events(control commands)are allowed to execute simultaneously,the supervisor in the framework of discrete event systems(DESs)selects one randomly.However,in practical applications,such as traffic scheduling and robot path planning,the problems of directed control and numerical optimization should be considered.This paper introduces an optimization mechanism to quantify the control cost and combines supervisory control theory(SCT)with reinforcement learning.A systematic procedure is proposed to synthesize the optimal directed supervisor of a DES based on reinforcement learning,which makes the controlled system achieve the following three goals:(1)the con-trol specifications relevant to security and liveness are not violated;(2)at most one controllable event can be executed at each state;(3)the cumulative cost of event execution from the initial state to a mark state is minimal.First,given the autom-aton models of the plant and specifications,the target automaton model is obtained by the synchronous operation of these two models;a cost function is defined and assigns the execution cost for each event in the target model.Second,the non-blocking and maximally permissive supervisor is synthesized by SCT.Finally,the supervisor is transformed into a Markov decision process and then the Q-learning algorithm is utilized to compute the optimal directed supervisor.Two applications are used to verify the effectiveness and correctness of the proposed method.The simulation results show that the proposed method can realize the directed control of the system,and the numerical cost of the directed supervisor is minimized.

discrete-event systemdirected supervisorreinforcement learningoptimal controlnumerical optimiza-tiontraffic systems

胡瑜洪、王德光、杨明、王玺

展开 >

贵州大学电气工程学院,贵州 贵阳 550025

西安电子科技大学机电工程学院,陕西 西安 710071

离散事件系统 定向监控器 强化学习 最优控制 数值优化 交通系统

国家自然科学基金国家自然科学基金贵州省省级科技计划资助项目贵州省教育厅青年科技人才成长项目贵州大学科研基金资助项目贵州省教育厅创新群体

5226506662203132黔科合基础-ZK[2022]一般103黔教合KY字[2022]138号贵大特岗合字[2021]04号黔科合支撑[2021]012

2024

10.12263/DZXB.20221267

电子学报
中国电子学会

电子学报

CSTPCD北大核心
影响因子：1.237
ISSN：0372-2112
年,卷(期)：2024.52(9)