Nadam

Combines NAG and Adam.

It uses look-ahead momentum vector directly to the current parameter update

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1)g_t$$ $$m'_t = \frac{m_t}{1 - \beta_1^t}$$ $$u_t = \beta_1 m'_{t-1} + \frac{(1 - \beta_1)g_t}{1 - \beta_1^t}$$ $$\Delta \theta_t = - \frac{\eta}{\sqrt{v'_t} + \epsilon} u_t$$ $$\theta_t = \theta_{t-1} + \Delta \theta_t$$

Where:

1. all other equations and parameters are as explained in Adam