Embeddings: meaning as geometry

An embedding is a row of a matrix

Let $V$ be the vocabulary size and $d$ the dimension of the vectors. The table is a matrix $E\in\mathbb{R}^{V\times d}$. The word with index $i$ is the one-hot vector $x_i\in\{0,1\}^V$ (a one at position $i$, zeros elsewhere) and its embedding is

Multiplying by a one-hot equals picking a row: it is a linear layer with no activation (lesson 7) whose $V\cdot d$ weights are learned. With $V=400{,}000$ and $d=100$ that is $40{,}000{,}000$ numbers. In practice nothing is multiplied: the row is looked up. The same mechanism opens language models: in the smallest GPT-2 the table is $50{,}257\times 768\approx 38.6$ million parameters (lessons 20 and 22).

Cosine similarity

If both vectors have length 1, $\cos(a,b)=a\cdot b$ and $\lVert a-b\rVert^2=2-2\cos(a,b)$: ranking by cosine and ranking by Euclidean distance gives the same order. Example: $a=(3,4)$ and $b=(4,3)$ give $\cos=\frac{12+12}{5\cdot 5}=0.96$, an angle of about $16^\circ$. With $\cos=0.88$ (“dog” and “cat”) the angle is $\arccos 0.88\approx 28^\circ$.

Where do the numbers come from?

Skip-gram (Mikolov et al., 2013) maximizes the average probability of the neighbors of each word $w_t$, with a window of size $c$:

Each word has an input vector $v$ (its embedding) and an output vector $v'$. The denominator runs over the whole vocabulary, which is why the paper proposes approximations such as negative sampling. GloVe (Pennington et al., 2014) instead fits

where $X_{ij}$ counts how often $j$ appears near $i$ in the corpus, trained with AdaGrad. Two roads, one idea: words that share contexts end up with similar vectors.

The arithmetic, step by step

The question “$a$ is to $a^*$ as $b$ is to ...” is answered with

With length-1 vectors, $\lVert a^*-a+b\rVert$ does not depend on $x$, so this equals maximizing $\cos(x,a^*)-\cos(x,a)+\cos(x,b)$. The exclusion is not a detail: Linzen (2016) measured that, without it, the closest word was $b$ in 93% of cases and $a^*$ in 5%. In this lesson, with GloVe’s 100 numbers and choosing among its 400,000 words:

Questions	With exclusion	Without exclusion	The closest was one of the 3 inputs
Capital–country (342)	80.1% (274)	68.1%	19.3%
Gender (90)	85.6% (77)	55.6%	41.1%

I picked the pairs, and they are clean relations; the Mikolov et al. set (19,544 questions, 14 types, counting only the exact word, not synonyms) is harder: their best model reached 65.6%. With only the lesson’s 880 words the exam is far easier (94.4% and 85.6%): there are fewer candidates to get wrong. Linzen adds that the method mixes the consistency of the offset with neighborhood structure: it does not measure “relations” alone.

How the bias is measured in the lesson

With $F$ = {woman, girl, mother, wife, sister, daughter} and $M$ = {man, boy, father, husband, brother, son}, the score of a word $w$ is

Positive means closer to the female words. Sets are used rather than a pair because “man” has several senses. It is a simplified version of the association tests of Caliskan et al. (2017). Result over the 52 professions in the vocabulary: from $-0.16$ (“boss”) to $+0.10$ (“receptionist”); 26 objects with no gender (table, tree, river…) fall between $-0.05$ and $+0.03$. The Spearman correlation with the “she” − “he” score in the full GloVe is 0.91. The score mixes gender with topic (in the text, the female words show up more next to “dress” or “soap”): it shows the slope, it is not a verdict about any person. If each vector has its component along $g=\bar F-\bar M$ (normalized) subtracted, the score on that axis is 0 by construction; even so, guessing whether each profession was above or below the median of the original score, with the nearest cosine centroid and leaving one out at a time, gets 41 of 52 right before and 35 of 52 after (chance: 26). It is a minimal experiment, with few words, in the spirit of Gonen and Goldberg (2019), who showed that methods for “debiasing” mostly hide the bias rather than remove it.

Semantic search

The average of a text with words $w_1,\dots,w_n$ is $q=\frac1n\sum_k e_{w_k}$; documents are ranked by $\cos(q,D_j)$. Comparing against $N$ documents costs $O(N\cdot d)$ per query, and with millions of them approximate indexes are used. Sentence vectors (Reimers and Gurevych, 2019) are computed once per document: finding the most similar pair among 10,000 sentences with BERT takes about 50 million inferences (about 65 hours); with sentence vectors, about 5 seconds. The third column of the scene uses paraphrase-multilingual-MiniLM-L12-v2 (Apache-2.0, 384 dimensions, more than 50 languages; Reimers and Gurevych, 2020), with the mean of the output vectors and cosine. Its scores were computed offline for the 36 test questions.

With 36 questions, 21 hits (the vector average on the English questions) are 58.3% with a 95% Wilson interval of 42% to 73%: a margin of about ±15 points. That is why lesson 34 insists on sample size.

In code

With toy vectors (invented numbers) the whole mechanism is visible:

import numpy as np

# Toy embedding table: one row per word, 3 numbers each (invented; a real model uses hundreds).
words = ["king", "queen", "man", "woman", "bread"]
E = np.array([
    [4.0, 3.0,  1.0],   # king
    [4.0, 0.5, -2.0],   # queen
    [4.0, 0.0,  1.0],   # man
    [4.0, 0.0, -1.0],   # woman
    [0.0, 0.5,  3.0],   # bread
])
E = E / np.linalg.norm(E, axis=1, keepdims=True)   # length 1: the dot product is the cosine
row = {w: i for i, w in enumerate(words)}

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

def closest(v, exclude=()):
    pairs = [(w, round(cosine(v, E[i]), 2)) for w, i in row.items() if w not in exclude]
    return sorted(pairs, key=lambda p: -p[1])[:3]

print(round(cosine(E[row["king"]], E[row["queen"]]), 2))          # king and queen
v = E[row["king"]] - E[row["man"]] + E[row["woman"]]               # king - man + woman
print(closest(v))                                                   # without excluding
print(closest(v, exclude=["king", "man", "woman"]))                 # excluding the three inputs
# 0.68
# [('king', 0.89), ('queen', 0.87), ('woman', 0.81)]
# [('queen', 0.87), ('bread', -0.18)]

And with the real GloVe vectors (the file glove.6B.100d.txt comes from the Stanford page, see the sources; reading it takes a few seconds):

import numpy as np

words, rows = [], []
with open("glove.6B.100d.txt", encoding="utf8") as f:
    for line in f:
        w, _, rest = line.partition(" ")
        words.append(w)
        rows.append(np.asarray(rest.split(), dtype=np.float32))
E = np.stack(rows)
E /= np.linalg.norm(E, axis=1, keepdims=True)   # length 1: the dot product is the cosine
idx = {w: i for i, w in enumerate(words)}

def closest(v, k=5, exclude=()):
    s = E @ (v / np.linalg.norm(v))
    for w in exclude:
        s[idx[w]] = -1
    top = np.argsort(-s)[:k]
    return [(words[i], round(float(s[i]), 2)) for i in top]

print(closest(E[idx["dog"]], 5, exclude=["dog"]))
v = E[idx["king"]] - E[idx["man"]] + E[idx["woman"]]
print(closest(v, 3))
print(closest(v, 3, exclude=["king", "man", "woman"]))
# [('cat', 0.88), ('dogs', 0.83), ('pet', 0.74), ('puppy', 0.72), ('horse', 0.71)]
# [('king', 0.84), ('queen', 0.77), ('monarch', 0.68)]
# [('queen', 0.77), ('monarch', 0.68), ('throne', 0.68)]

Where this lesson’s data come from

Vectors: GloVe 6B with 100 dimensions (Wikipedia 2014 and Gigaword 5, 6 billion words, 400,000 vocabulary, English only), published by Stanford under the Public Domain Dedication and License v1.0, which allows redistribution. They are normalized to length 1 and stored in 8 bits (maximum cosine error 0.0095). The map is t-SNE (van der Maaten and Hinton, 2008) with perplexity 20, cosine metric, and seed 0: Wattenberg et al. (2016) warn that in a t-SNE the size of clusters and the distance between them may mean nothing. The 10 support articles and the 36 test questions were written by the lesson’s author: they are illustrative. Everything can be regenerated with the scripts in the lesson folder (build-vectors.py, eval-search.py, eval-analogies.py, measure-data.py).

What comes next

The vectors in this lesson are static: one word, one vector. Attention (lesson 21) and the transformer (lesson 22) produce a different vector for each word depending on the sentence. Lesson 15 asks whether, besides representing what exists, a network can generate something new; lesson 18 puts image and text vectors in one shared space, and lesson 31 uses semantic search to give a model knowledge of its own.