RMSProp

Same as Adagrad, but, the running average is limited to a fixed window. Developed around the same time as that of AdaDelta.

• $$E[g^2]_t = \gamma E[g^2]_{t-1} + (1 - \gamma) g_t^2$$
• $$RMS[g]_t = \sqrt{\epsilon + E[g^2]_t}$$
• $$\Delta \theta_t = - \frac{\eta}{RMS[g]_t} g_t$$
• $$\theta_t = \theta_{t-1} + \Delta \theta_t$$

Where:

1. all other equations and parameters are as explained in Adagrad