Why Computer Vision Is Hard
If images are "just numbers," why is computer vision so difficult? The answer is that the gap between raw pixel values and semantic meaning is enormous, and the real world adds noise to that gap at every turn.
The Messiness Problem
A model trained in a clean, well-lit laboratory environment will often fail when deployed in the real world. Lighting changes. Backgrounds change. Objects get partially hidden by other objects (occlusion). People tilt their phones. Shadows appear. Kids and pets run through the frame.
From the Field: Synthetic Data to the Rescue
In one fitness application, a pose detection model performed beautifully in a test setup — controlled lighting, plain background, unobstructed views. The moment real users opened the app, it turned out that apartments contain furniture, pets, and dim evening light. The training data contained none of those things.
The fix was a synthetic data generation effort: thousands of artificial human figures in hundreds of different environments, poses, and lighting conditions. The resulting models were substantially more robust because they gave the model exposure to the kinds of variation it would encounter in the wild.
The Spatial Reasoning Problem
Images are inherently 2D projections of a 3D world. Depth information is lost. Relative positions of objects can be ambiguous. Anyone who has ever been fooled by an optical illusion has experienced first-hand how difficult spatial reasoning from 2D images can be, and machines face the same challenge with far less contextual background knowledge than humans bring.
The Data and Speed Problem
Computer vision models require very large, carefully labeled datasets. Labeling images is slow and expensive. The scale required — often hundreds of thousands or millions of images — means that dataset construction is frequently the bottleneck, not the modeling.
At the same time, many of the most important applications require extremely fast inference. A self-driving car cannot afford a two-second response time. This tension between model accuracy (which improves with scale and complexity) and inference speed (which demands efficiency) is one of the defining trade-offs of the field.
A factory inspection model achieves 99% accuracy on the validation set but misses defects when the production line lighting shifts at night. Which computer vision challenge does this best illustrate?