Adamax

Same as Adam, but uses the $l_{\infty}$ norm in the running average of past gradients.

$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) |g_t|^\infty$$

OR

$$v_t = max(\beta_2 v_{t-1}, |g_t|)$$

Where:

1. all other equations and parameters are as in Adam