Embeddings

1 min read
On trails:AGRAGstep 2/8

An embedding is an array of numbers that stands for the meaning of a piece of data, something like [0.91, 0.15, 0.83, ...]. Representing data as vectors captures the semantic nature of unstructured data (text, images), which keyword matching cannot.

Meaning as distance

An embedding model turns text into a vector so that texts with similar meaning land close together. Searching by meaning then becomes a distance calculation. Cosine similarity is the usual one.

Python
from math import sqrt def cosine(a, b):    dot = sum(x * y for x, y in zip(a, b))    return dot / (sqrt(sum(x * x for x in a)) * sqrt(sum(y * y for y in b))) # Toy vectors with three numbers. Real embeddings have hundreds or thousands.refund = [0.91, 0.15, 0.83]money_back = [0.88, 0.20, 0.79]weather = [0.05, 0.93, 0.10] print(cosine(refund, money_back))  # close to 1: similar meaningprint(cosine(refund, weather))     # much lower: unrelated

Similar meaning scores near 1, unrelated meaning scores lower. The vectors here are made up and tiny, to show the idea only.

Embedding models

ModelEmbedsHow it learns
Word2VecWordsCBOW predicts a word from its context, skip-gram predicts the surrounding words from a word
GloVeWordsGlobal word-word co-occurrence counts over a corpus
CLIPImages and textImage-caption pairs, so both land in one shared space

These are the classic examples. Word2Vec and GloVe give one vector per word, while RAG over documents needs a model that embeds a whole chunk. The query and the chunks must be embedded by the same model, or the distances mean nothing. The vectors are stored and searched in a vector databaseVector databasesStores for embeddings that find the nearest ones fast, by trading a little accuracy for speed.Opens in a new tab.

Sources