Loss Functions

To improve, you need to know how wrong you are. That is the job of the loss function (also called a cost function, though technically the loss function applies to a single example while the cost function is the average loss across the entire training set — in practice, these terms are used interchangeably).

Consider a simple example: you are training a model to count blueberries in a muffin from an image. Your model looks at the image and predicts 2 blueberries. The actual label is 4. Your error is 2. The loss function formalizes and quantifies that error.

Common Loss Functions for Regression

• Mean Squared Error (MSE)

  $$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n}(y_i - \hat{y}_i)^2$$

  Squares the error, so large errors are penalized heavily. Use this when outlier errors are genuinely costly — house price prediction, energy consumption forecasting.

• Mean Absolute Error (MAE)

  $$\text{MAE} = \frac{1}{n} \sum_{i=1}^{n}|y_i - \hat{y}_i|$$

  Takes the absolute value of the error, treating all errors proportionally. Use when outlier errors should not dominate training — food delivery time prediction, where one delayed package should not distort the entire model.

• Huber Loss

  $$L_\delta(y, f(x)) = \begin{cases} \frac{1}{2}(f(x) - y)^2 & \text{if } |f(x) - y| \leq \delta \\ \delta \left|f(x) - y\right| - \frac{1}{2}\delta^2 & \text{if } |f(x) - y| > \delta \end{cases}$$

  A hybrid that behaves like MSE for small errors and MAE for large errors. More robust than MSE but adds a threshold hyperparameter $delta$.

Common Loss Functions for Classification

• Binary Cross-Entropy

  $$L(y, f(x)) = -\left[y \log(f(x)) + (1 - y) \log(1 - f(x))\right]$$

  Used when you have exactly two classes (spam/not spam, dog/not dog). Measures the divergence between the predicted probability distribution and the true distribution.

• Categorical Cross-Entropy

  $$L(y, f(x)) = -\sum_{i} y_i \log(f(x)_i)$$

  Used when you have more than two classes. Generalizes binary cross-entropy to the multi-class case.