An adaptive variance reduction method with negative momentum
Stochastic variance reduction methods have been successful in solving large scale machine learning problems,and researchers cooperate them with adaptive stepsize schemes to further alleviate the burden of parameter-tuning.In this article,we propose that there exists a trade-off between progress and effectiveness of adaptive stepsize arising in the SVRG-BB algorithm.To enhance the practical performance of SVRG-BB,we introduce the Katyusha momentum to handle the aforementioned trade-off.The linear convergence rate of the resulting SVRG-BB-Katyusha algorithm is proven under strong convexity condition.Moreover,we propose SVRG-BB-Katyusha-SPARSE algorithm which uses Katyusha momentum sparsely in the inner iterations.Numerical experiments are given to illustrate that the proposed algorithms have promising advantages over SVRG-BB,in the sense that the optimality gaps of the proposed algorithms are smaller than the optimality gap of SVRG-BB by orders of magnitude.
万方数据
维普
