Stochastic Gradient Descent

This performs gradient descent on each of the samples in the dataset.

• $$g_t = \frac{\partial J(\theta_t, x^i, y^i)}{\partial \theta_t}$$

Where:

1. $x^i, y^i$ = i’th input sample and label in the dataset
2. all other equations and parameters are as explained in gradient descent

Pros:

1. Runs fast
2. Similar convergence guarantees as in batch gradient descent, though one should gradually anneal $\eta$ over time to achieve this.
3. Supports online learning

Cons:

1. Creates high variance updates to the parameters