AdaDelta

Same as Adagrad, but, the running average is limited to a fixed window.

Moving window average is defined as:

• $$E[x^2]_t = \gamma E[x^2]_{t-1} + (1 - \gamma) x_t^2$$

The updates for AdaDelta will then be as follows:

• $$\Delta \theta_t = \frac{\eta}{\sqrt{\epsilon + E[g^2]_t}} g_t$$
• $$E[\Delta \theta^2]_t = \gamma E[\Delta \theta^2]_{t-1} + (1 - \gamma) \Delta \theta_t^2$$
• $$RMS[\Delta \theta]_t = \sqrt{\epsilon + E[\Delta \theta^2]_t}$$
• $$\Delta \theta_t = -\frac{RMS[\Delta \theta]_{t-1}}{RMS[g]_t} g_t$$
• $$\theta_t = \theta_{t-1} + \Delta \theta_t$$

Where:

1. all other equations and parameters are as explained in Adagrad