Latent space: the hidden meaning in numbers

In What are embeddings? we turned each word into a list of 300 numbers and saw that similar words end up close together. The space those vectors live in has a name: a latent space. “Latent” means hidden. Nobody designed its coordinates; they were learned, and their meaning has to be discovered after the fact. This article is about what that hidden structure looks like, what you can read from it, and where it misleads.

What “latent space” means

Any model that turns an input into a vector of numbers defines a latent space: the set of all vectors it could produce. Word embeddings are the classic example, but the same word is used for the internal vectors of image generators, speech models and large language models. The key property is that position encodes something about the input. Two inputs the model treats as similar land near each other.

Everything below uses real word vectors (GloVe, 300 dimensions, 10,000 common English words). The ideas carry over to other latent spaces, even when the details differ.

The axes don’t have names

A natural first guess is that each of the 300 numbers measures one thing: dimension 12 for “is an animal”, dimension 40 for “is royal”. In practice, individual dimensions are almost never that tidy.

The reason is simple. Training only cares about how vectors relate to each other: their distances and angles. If you rotated the whole space, every distance and angle would stay the same, and the model would work exactly as well. So nothing pushes a concept to line up with one particular axis. Meaning ends up spread across many dimensions at once.

That is why reading an embedding one number at a time tells you very little. Meaning lives in directions, and a direction is a particular mix of all 300 numbers.

Directions can carry meaning

Take the arrow from man to woman: the difference between their two vectors. Now take the arrow from king to queen. If the space has learned something consistent about gender, those arrows should point roughly the same way. They do, and so do the arrows for boy→girl, uncle→aunt and actor→actress.

“Roughly” is doing real work in that sentence. A cosine of 0.58 means the arrows share a common direction, but each also has its own quirks. Other relationships are weaker still. The singular-to-plural arrows (cat→cats, city→cities, child→children and so on) agree with each other much less, with an average cosine of only 0.23 in this data.

Analogies as parallel arrows

If an arrow means “make it female”, you can add it to other words. This is the famous analogy trick: king − man + woman lands near queen. The same arithmetic finds capitals (paris − france + italy ≈ rome) and comparatives (bigger − big + small ≈ smaller).

Two honest caveats. First, the result vector is usually still closest to one of the words you started with, so analogy tests quietly skip the input words. Second, it is easy to show the analogies that work and forget the ones that don’t. The trick is a real sign of structure, not a reliable reasoning engine. The site guide king − man + woman lets you try your own.

Neighbourhoods and clusters

The simplest structure in a latent space is the neighbourhood. Colours sit near colours, numbers near numbers, capital cities near their countries. Algorithms such as k-means can find these groups automatically by looking for dense clumps of points, which is how embeddings are used to group documents or customer reviews by topic.

Clusters are not labelled, and their edges are fuzzy. A word with several senses sits between groups: apple sits between fruit and technology companies. Whatever you call a cluster is your interpretation, not something stored in the vectors.

Walking between two points

If positions mean something, what about the space between two words? You can blend two vectors, say 70% village and 30% city, and ask which real words are nearest to the blend.

Two things show up. The honest one first: at every step, one of the two end words is still the closest real word. There is no hidden “half village, half city” word waiting in the middle. But the runner-up words change in a sensible way. Between village and city you pass town; between hot and cold you pass cool and warm; from king to queen you pass monarch. The space is smooth enough that points between two meanings are near words that share both.

In image generators this smoothness is put to work. Many generative models have latent spaces where the points between two images decode to believable in-between images, which is how smooth “morphing” animations are made.

Not just words

Latent spaces exist wherever a model turns data into vectors. Sentence embedding models map whole sentences to one vector each, so “How do I reset my password?” can land near “I forgot my login” even though the two share no words. Image models map pictures to vectors. Some models, such as OpenAI’s CLIP, are trained so that a photo and a caption describing it land near each other in one shared space, which is what makes searching photos by text possible. See Tokens beyond text for how images and audio are cut into pieces first.

Inside a large language model there isn’t one latent space but many: every layer produces new vectors for every token (see Attention and the transformer). Researchers find meaningful directions there too, but reading them is an active research area, not a solved problem.

Limits and bias

A latent space is a compressed record of its training data, including the parts we might not want. Words like nurse or engineer carry no gender in their definitions, but they appeared in gendered contexts often enough that their vectors lean one way.

This was documented carefully in a 2016 paper whose title quotes a real analogy result: “Man is to Computer Programmer as Woman is to Homemaker?”. In our data, doctor − man + woman gives physician first and nurse second. Any system built on top of embeddings, from search to hiring tools, can pass these patterns on. Techniques exist to reduce them, but none removes them completely.

One more limit is on our side. We can only look at these spaces through flat pictures, and squashing 300 dimensions into two always distorts something. Seeing high dimensions explains how those pictures are made and how to read them without being fooled. To explore GloVe yourself, try the Embedding explorer or do word arithmetic in the Vector Playground.