Abstract
In the paper,we investigate the optimization problem(OP)by applying the optimal control method.The optimization problem is reformulated as an optimal control problem(OCP)where the con-troller(iteration updating)is designed to minimize the sum of costs in the future time instant,which thus theoretically generates the"optimal algorithm"(fastest and most stable).By adopting the maximum prin-ciple and linearization with Taylor expansion,new algorithms are proposed.It is shown that the proposed algorithms have a superlinear convergence rate and thus converge more rapidly than the gradient descent;meanwhile,they are superior to Newton's method because they are not divergent in general and can be ap-plied in the case of a singular or indefinite Hessian matrix.More importantly,the OCP method contains the gradient descent and the Newton's method as special cases,which discovers the theoretical basis of gradient descent and Newton's method and reveals how far these algorithms are from the optimal algorithm.The merits of the proposed optimization algorithm are illustrated by numerical experiments.
维普
万方数据