The Problem We Haven't Solved
Word2Vec was a real leap. It gave us dense, semantically meaningful embeddings, and even let us do arithmetic on them. But each word has one vector forever. bank is bank. Static.
What we want is for bank in "bank of the river" to end up close to embeddings about rivers and shores, while bank in "deposited money at the bank" ends up close to embeddings about finance. Same word, different vectors, dictated by context.
ℹ
Part II Begins Here
Everything in Part I gave us tools to represent language that are better than nothing. But they all share one fundamental limitation: representations don't depend on context. This chapter introduces the mechanism that finally solves that — and in doing so, redirected the entire field.