Stochastic gradient descent algorithm based on ensemble importance sampling
Many machine learning and deep learning problems can be solved using stochastic gradient optimization algorithms.Most of the popular algorithms currently use uniform sampling to extract samples from a dataset to calculate gradient estimates.However,randomly sampled gradient estimates will result in a high variance,which will accumulate as the optimization proceeds,reducing the convergence speed of the algorithm.To address this issue,different sampling probabilities can be assigned to each sample.This paper proposes a new algorithm for selecting non-uniform sampling distributions based on the idea of ensemble learning.The primary goal of the algorithm is to select a sampler weight such that the variance of the gradient estimate is minimized.The proposed algorithm consists of multiple simple samplers,and the sampling weight assigns a contribution weight to each simple sampler to obtain the final sampling distribution.The ensemble importance sampling algorithm can be combined with previous stochastic gradient optimization methods.This paper presents a stochastic gradient descent algorithm using ensemble importance sampling.In the experiments,the effectiveness of the algorithm is demonstrated.The proposed algorithm reduces the variance in real datasets which has certain advantages compared to other algorithms.
万方数据
维普
