Why We Need a Better Representation

Bag of words on a real corpus is brutally inefficient. The standard English dictionary has roughly 470,000 entries. If every dimension of your vector corresponds to a word, you're working in 470,000-dimensional space, and almost every dimension is zero almost all the time. Sparse, huge, and semantically uninformative. The dimensions for cat and dog are no more related than the dimensions for cat and pneumonia.

What we want instead is to take that word bank and map it into a much smaller space, say 300 or 512 dimensions, where the dimensions themselves capture something about meaning. Where cat and dog are close together. Where bank (financial) is near money and loan, and bank (river) is near shore and stream.

This is what neural networks are great at. We can train a network to produce a dense, low-dimensional encoding for each word that captures the word's semantic role.