Consider stochastic gradient descent with momentum,
mt+1=βmt+(1−β)gt,wt+1=wt−γmt+1,with 0≤β<1. The stochastic gradients gt are i.i.d. with E[g]=μ and .
In the steady-state limit t→∞, the variance of the momentum term converges to
Var[m∞]=1+β1−βσFor β=0.8, compute Var[m∞].