Word Embeddings

Word Embeddings

Word embeddings are dense, continuous vector representations of words in a low-dimensional space. Unlike sparse BoW or TF-IDF vectors, embeddings capture semantic relationships—words with similar meanings have similar vectors.

Word Embedding Methods Comparison

Using Pre-trained Embeddings

import numpy as np
from gensim.models import KeyedVectors

# Load pre-trained Word2Vec
model = KeyedVectors.load_word2vec_format(
    'GoogleNews-vectors-negative300.bin', binary=True
)

# Similarity
print(model.similarity('king', 'queen'))  # ~0.65
print(model.similarity('king', 'man'))    # ~0.32

# Most similar
print(model.most_similar('computer', topn=5))
# [('computers', 0.87), ('software', 0.78), ...]

# Analogy: king - man + woman = queen
result = model.most_similar(
    positive=['king', 'woman'],
    negative=['man'],
    topn=1
)
print(result)  # [('queen', 0.71)]

Cosine Similarity for Embeddings

def cosine_sim(v1, v2):
    return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))

king = model['king']
queen = model['queen']
man = model['man']

print(f"king-queen: {cosine_sim(king, queen):.3f}")
print(f"king-man: {cosine_sim(king, man):.3f}")

Visualizing Embeddings

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

words = ['king', 'queen', 'man', 'woman', 'prince', 'princess',
         'cat', 'dog', 'fish', 'bird']

vectors = np.array([model[w] for w in words])
tsne = TSNE(n_components=2, random_state=42)
coords = tsne.fit_transform(vectors)

plt.figure(figsize=(10, 8))
plt.scatter(coords[:, 0], coords[:, 1], c='red', alpha=0.7)
for i, word in enumerate(words):
    plt.annotate(word, (coords[i, 0], coords[i, 1]))
plt.title("Word Embeddings Visualization")
plt.show()

Properties of Word Embeddings

• Semantic similarity: Similar words have close vectors
• Linear relationships: king - man + woman ≈ queen
• Clustering: Semantically related words form clusters
• Compositionality: Can combine vectors for phrases