The Limitation of Word Embeddings
Word2Vec embeddings are static. Each word has exactly one vector. So bank has one vector, regardless of whether you're talking about a river or a financial institution. The vector ends up somewhere in between, a kind of weighted compromise that satisfies neither meaning fully.
We haven't fixed bank of the river versus deposited money at the bank. We've made it less bad, but the contextual problem is still very much open.
What's Still Missing
Static embeddings have no concept of context within a sentence. The embedding for orange in "I ate an orange" and in "The sunset is orange" is the same vector — a compromise between the fruit and the color. The word needs a dynamic, context-dependent representation.
Real World: Recommendation and Classification at Scale
Pre-trained Word2Vec and GloVe embeddings are still production-grade features for many search, recommendation, and classification pipelines where the computational budget for transformers isn't justified. When recommending products based on product names, or routing customer service tickets to teams, Word2Vec embeddings + logistic regression is often the right answer.
A production search system uses pre-trained Word2Vec embeddings to represent product titles and customer queries. A user searches for 'running shoes' but the most relevant results are labeled 'athletic footwear.' The system returns poor results. Why?