Anchor Boxes and Non-Maximum Suppression

The Anchor Box Problem

Here is the challenge in bounding box detection: objects come in wildly different sizes, shapes, and positions. A naive approach might slide a fixed-size window across every position in the image. But what size window? A window sized for a distant pedestrian will miss a nearby truck, and vice versa.

The solution is anchor boxes: a predefined set of boxes with different sizes and aspect ratios, placed at every position in the image. The model does not predict bounding boxes from scratch — it predicts offsets (adjustments to x, y, width, and height) relative to each anchor box. This is a more tractable prediction problem.

A typical detection network might use thousands of anchor boxes per image. That raises a new problem: how do you deal with all those predictions?

Non-Maximum Suppression: Keeping the Best Boxes

When a model generates thousands of anchor box predictions, two things happen: many boxes will have very low confidence (nothing detected), and many remaining boxes will overlap heavily (multiple anchors detecting the same object from slightly different positions).

Non-Maximum Suppression (NMS) handles both:

1. Remove all boxes with confidence below a threshold (e.g., 80%).
2. For the remaining boxes, calculate Intersection over Union (IoU) between every pair.
3. When two boxes have IoU above a threshold — they are clearly detecting the same object — keep only the one with higher confidence and discard the other.

Intersection over Union is the area of overlap between two boxes divided by the area of their union. An IoU of 1.0 means perfect overlap. An IoU near 0 means the boxes barely share any space.