CHAPTER 8.2c · EMBEDDINGS

Word Analogies through Vector Arithmetic

Embeddings encode semantic relationships: "king - man + woman = queen". Analogies can be solved through simple vector addition and subtraction.

Word Analogies demonstrate the remarkable property of embeddings: Semantic relationships become geometric relations. "king - man + woman = queen" works because gender and royalty are encoded as directions in vector space.

📖 Learning Context ▼
🎯

Learning Objectives

  • Apply vector arithmetic to embeddings
  • Understand semantic directions in space
  • Learn the limitations of analogies
🗺️

Context: Where are we?

Step 4/5 Chapter 8: Tools & Glossary

Embeddings & Tokens (4/5) – the geometric interpretation of meaning.

💡

Why It Matters

Analogies prove that embeddings learn real semantics – not just word frequencies. This is the foundation for transfer learning and zero-shot generalization.

📌

Key Takeaways

  • Vector Addition: a - b + c ≈ d for analogies
  • Directions: Gender, tense, size are vectors
  • Limitations: Not all relationships are linearly encoded
Choose an analogy:
Nearest Neighbor (Result)
queen
Cosine Similarity: 0.94
How Does It Work?
Embeddings learn semantic relationships during training. "king" and "queen" have similar vectors, but the difference to "man"/"woman" encodes gender. These difference vectors are consistent.
Vector Arithmetic
result = E["king"] - E["man"] + E["woman"]. We compute the nearest neighbor to this result vector (via cosine similarity), often it's the expected word.
Limitation
These analogies don't always work perfectly. They depend on training data quality and can reflect biases. Modern contextual embeddings (BERT, GPT) are more complex.
Historical Significance
Word2Vec (2013) made these analogies famous. They showed that neural embeddings learn semantic structure – a breakthrough in NLP.