Gradient Descent

The goal of machine learning is to use data to minimize some particular objective (loss) function. The most widespread way to do this is through gradient descent, which essentially takes small steps in the direction of steepest descent for a particular objective function to find the “minimum”.

$$x_{t+1}=x_t-{\alpha }_t\nabla f(x_t)$$

• $\alpha$ = learning rate

In practice, we use stochastic gradient descent (SGD), which uses one data point at a time for a single step and uses a much smaller subset of data points at any given step.

Method	Data Used per Update	Update Rule	Speed	Stability	Noise	Best Used When
Vanilla GD / Batch GD	All training data $\{(x_i,y_i){\}}_{i=1}^N$	$\theta =\theta -\alpha \cdot \frac{1}{N}{\sum }_{i=1}^N{\nabla }_{\theta }{\mathcal{L}}_i(\theta )$	Slow	Very stable	No noise	Small datasets
Stochastic GD (SGD)	One sample $(x_i,y_i)$	$\theta =\theta -\alpha \cdot {\nabla }_{\theta }{\mathcal{L}}_i(\theta )$	Fast	High variance	High noise	Large datasets or online learning
Mini-batch GD	A small batch of $m\ll N$ samples	$\theta =\theta -\alpha \cdot \frac{1}{m}{\sum }_{j=1}^m{\nabla }_{\theta }{\mathcal{L}}_j(\theta )$	Medium	Good tradeoff	Medium noise	Deep learning (standard default)