Data Augmentation
Even with transfer learning, the quantity and diversity of training data matters. Real-world images vary in lighting, orientation, scale, and position. If your training data does not reflect that variation, your model will be brittle.
Data augmentation artificially introduces variation into your training data by randomly transforming images before they are fed to the model. At each epoch, different random transformations are applied, so the model effectively sees a different version of each image every time.
Critical Rule: Augment Training Data Only
Augmentation is applied only to training data. Validation and test sets should reflect the real-world distribution you are trying to evaluate on — they should not be artificially distorted. Augmenting your test set would give you a misleading metric that doesn't correspond to real performance.
Common Augmentation Transforms
- Random cropping — Present different subregions of the image. Forces the model to handle objects at different positions within the frame.
- Horizontal and vertical flipping — Useful for tasks where orientation is not meaningful (e.g., detecting tumors in scans).
- Random rotation — Helps the model handle objects that appear at unusual angles.
- Color jitter — Randomly adjust brightness, contrast, saturation. Prepares the model for different lighting conditions.
- Gaussian blur — Reduces reliance on high-frequency texture detail that may not appear consistently in deployment.
- Perspective and shear transforms — Simulate different camera angles and distances.
Toggle transforms and tune their strength. The preview updates across 6 independently sampled versions of the same image.
50% chance of mirroring left–right
Rotate by a random angle within ±max°
Randomly crop and resize to full size
Randomly shift brightness, contrast, saturation, hue
Original → 6 augmented samples
Original
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
Sample 6
Select augmentation transforms and adjust their parameters. See the same training image transformed in real time across multiple samples — the variation your model will see during training.
Real World: Synthetic Data Generation
For some applications, even augmented real data is not enough. One fitness app team generated entirely synthetic humans in artificial environments. The rendered humans were not photorealistic, but they exposed the model to enough variation in background, lighting, and occlusion that real-world performance improved substantially.
This approach — generating synthetic training data — is becoming increasingly common, especially in domains where real labeled data is expensive or ethically complex to collect. Diffusion models (covered in the final chapter) are now used for exactly this purpose.
You are training a model to detect defects on a production line. Your training set has 2,000 images of defective parts and 2,000 of good parts — all photographed under the same controlled lighting. The model performs well in testing but fails in production during the night shift. What augmentation strategy would you apply, and why?