Embeddings on AIBussin — AI applications, systems and books

What Is an Embedding?

Mon, 07 Sep 2026 09:00:00 +0000

Part I — A Vector Is Not Meaning

Three words and a list of numbers

Take three words:

cat
dog
airplane

Turn them into numbers. Any embedding API will do it. You get three arrays, each maybe 384 or 768 or 1,536 floats long:

cat [ 0.021, -0.114, 0.062, ... ]
dog [ 0.019, -0.098, 0.071, ... ]
airplane [-0.087, 0.203, -0.041, ... ]

Compute the angle between cat and dog. It is small. Compute the angle between cat and airplane. It is larger. Something about “cats and dogs are both pets” appears to have survived the trip into number-space.

Meaning Becomes Geometry

Mon, 07 Sep 2026 09:10:00 +0000

Part I — A Vector Is Not Meaning

A space you can draw

Give six words two coordinates each, by hand:

 x = monarchy / institutional power y = gendered association (-1 … +1)
king ( 0.9, 0.6)
queen ( 0.9, -0.6)
duke ( 0.7, 0.5)
duchess ( 0.7, -0.5)
apple (-0.7, 0.1)
orange (-0.7, -0.1)

Now several semantic questions have a geometric form:

Which words form a group? → apple and orange sit almost on top of each other, about 0.2 apart, and far from the four titles. The titles form a looser group of their own near x ≈ 0.8. Grouping shows up as density.
What distinguishes king from queen? → the vector king − queen ≈ (0, 1.2). It points almost purely along y. The difference between them, in this space, is one attribute. (duke − duchess ≈ (0, 1.0) points the same way, but only because we placed the points that way; a learned space owes us no such consistency.)
Is apple oriented like king? → the angle between them is obtuse and their cosine is negative. Fruit and titles point into different half-planes. Orientation separates the two categories cleanly, even though, as we are about to see, it is noisier within a category.
Are king and queen related? → obviously: they are the two monarchs. But their Euclidean distance is 1.2, while king sits 0.22 from duke and about 1.12 from duchess. Rank king’s neighbors by distance and queen comes third, behind duke and duchess. The gender axis — which a search for “royal titles” need not care about at all — has quietly reordered the neighborhood.

That last point is the one to carry forward.

Learning an Embedding Space

Mon, 07 Sep 2026 09:20:00 +0000

Part I — A Vector Is Not Meaning

Refusing the handout

For two chapters the vectors were given. Chapter 1 took them from an API and asked what was in them; Chapter 2 placed some by hand and treated the rest as the output of a model we agreed not to open. Chapter 2 closed on the question it left unanswered: where does the geometry come from?

This chapter builds a small space from nothing but a corpus and a counting rule. A usable geometry appears — related words land near each other without anyone labeling an axis “animal” or “drink.” Then comes the more important half: we inspect the construction closely enough to predict, from the mechanism alone, what it cannot preserve. When the measured results arrive, the failures are not surprises. They are consequences of the path the information took.

Similarity Is a Decision

Mon, 07 Sep 2026 09:30:00 +0000

Part I — A Vector Is Not Meaning

Two vectors, four answers

Here are three document vectors, kept to three dimensions so every number is checkable:

x = ( 2.0, 0.0, 0.0 ) a short doc, one strong topic
y = ( 6.0, 0.1, 0.0 ) a long doc, same topic, much more of it
z = ( 0.0, 2.0, 0.0 ) a short doc, a different topic

Before computing anything, look at the geometry. x and y point in almost exactly the same direction but have very different lengths (norm 2 versus about 6). x and z have the same length but point ninety degrees apart. So “is x more like y or z?” comes down to a prior question: which kind of difference should count — a difference in direction, or a difference in size?

Dimensions Do Not Mean What You Think

Mon, 07 Sep 2026 10:00:00 +0000

Part II — Inside the Space

What does dimension 173 mean?

Pull the 173rd coordinate of every vector in your corpus and sort. You get a list of documents ordered by something. Once in a while a coordinate is weakly readable — “this one runs high for questions” — but usually the sorted list has no nameable theme. Dimension 173 is not “formality,” not “sentiment,” not “is about sports.”

And yet the vectors work. Similarity search returns sensible results, clusters fall out, nearest neighbors are reasonable. The usable structure is there. It is just not sitting on the individual axes.

Neighborhoods and Hubs

Mon, 07 Sep 2026 10:10:00 +0000

Part II — Inside the Space

The question a vector cannot answer alone

Hand someone a single embedding vector — [0.13, -0.82, ...] — and ask what it means. They cannot say. Chapter 5 is why: the basis is not identifiable, the scale is model-specific, no coordinate names a concept.

Hand them the same vector and its ten nearest neighbors — all customer-support apologies, all in the same register — and they can usually name the topic, the register, and roughly what the point is about.

How Many Dimensions Does Meaning Need?

Mon, 07 Sep 2026 10:20:00 +0000

Part II — Inside the Space

A representation with several sizes

all-mpnet-base-v2 emits vectors of length 768. Embed the 1,173 RELATE items with it and ask several different questions about the size of the resulting cloud, and you do not get one answer:

nominal dimension 768
centered matrix-rank ceiling 768 (= min(1,173−1, 768))
95%-of-variance dimension 205
participation ratio 67.0
entropy effective rank 387.4
intrinsic-dimension estimate (MLE, k=10) 6.81
intrinsic-dimension estimate (TwoNN) 4.23

The Wave 2 dimensionality artifact records the last five numerical summaries plus nominal dimension; it does not persist an observed matrix-rank value, so the line above reports only the algebraic ceiling. Same vectors, same corpus: the reported or implied notions of size still range from about 4 to 768 — almost two orders of magnitude.

Similarity Is Not Discovery

Fri, 25 Sep 2026 02:45:00 +0100

The sidecar of Chapter 7 has an embarrassing regular guest. A researcher reading a paper on retrieval evaluation opens the side panel and finds, ranked first, a paper with nearly identical vocabulary — same benchmarks named, same metrics discussed, same dataset family. She opens it. It contributes nothing: same conclusions, weaker experiments, no new evidence, no disagreement, no extension. Topically it is the closest thing in the corpus to what she is reading. Informationally it is empty. The sidecar did its retrieval job perfectly and its discovery job not at all.

Feature Space: What Does a Linear Model Actually See?

Fri, 28 Aug 2026 15:30:00 +0100

Here is a sequence classifier. Eight examples, 128 positions each, 768 features per position, and a linear head that produces one score.

x = torch.randn(8, 128, 768)
classifier = nn.Linear(768, 1)

scores = classifier(x)

input (8, 128, 768)
Linear(768,1) -> (8, 128, 1)

One thousand and twenty-four scores where eight were expected. Nothing raised, nothing is non-finite, and the shape is entirely predictable once you know the rule. The layer did exactly what the tensor asked of it: nn.Linear transforms the last axis and preserves every axis before it, so it produced one score for every (example, position) pair — 128 scores per example, computed independently.

Hard Negatives

Mon, 07 Sep 2026 11:20:00 +0000

Part III — Retrieval Is an Experiment

Same model, four negative sets, four verdicts

MEASURED on RELATE v0.1, Wave 1 row 1.8 — artifact experiments/embeddings-from-first-principles/wave1/artifacts/margin-collapse.json. Model all-mpnet-base-v2, unchanged across every row below.

negative-selection rule mean margin selected-positive win rate
same-domain random, up to 5 per query 0.4738 1.0000
in-model selector, as implemented 0.2094 0.9814
structured perturbations 0.0932 0.9331
BM25 lexical top-5 0.0595 0.7621

The model did not change. The 269-query set did not change. The reference-positive rule did not change. Only the rule that picked which passages the positive had to beat changed — and the mean margin ranged from 0.4738 to 0.0595, a nearly eightfold difference between the widest and narrowest measured conditions.

Retrieval Is a Policy

Mon, 07 Sep 2026 11:30:00 +0000

Part III — Retrieval Is an Experiment

“Just retrieve the relevant documents”

There is no such operation. Chapter 11 closed on a single insight: a score never travels alone — it arrives with a negative-selection rule that is part of the measurement. This chapter widens the frame. A retrieval result never travels alone either. It arrives wrapped in a query construction, a representation, a candidate-eligibility rule, a scoring and fusion policy, a selection rule, an execution strategy, and a context-assembly step that decides what a downstream model actually sees.

Calibration

Mon, 07 Sep 2026 12:10:00 +0000

Part IV — Measuring the Representation

What is 0.81?

Illustrative scenario — invented values for intuition, not a measured result.

A pipeline decides two documents are “duplicates” if their cosine exceeds 0.8. Someone picked 0.8 because it looked reasonable. Imagine the two populations that number is actually sitting between:

distribution of cosine for KNOWN duplicate pairs: mean ~0.79
distribution of cosine for KNOWN non-duplicate pairs: mean ~0.71

If those two populations overlap the way this sketch implies, the threshold sits inside both: a true duplicate at 0.78 is rejected, an unrelated pair at 0.82 is accepted. This chapter’s actual experiment, later, replaces the sketch with measured positive and negative score samples and derived discrimination/operating statistics. The artifact does not persist the full distributions themselves, and its 85.6% figure is the coverage of a specific quantile-derived band rather than a generic measure of distributional overlap.

Is Similarity One-Dimensional?

Mon, 07 Sep 2026 12:20:00 +0000

Part IV — Measuring the Representation

Two pairs, same cosine, different situations

Illustrative vignette — constructed for intuition, not a measured RELATE example.

Consider what a retrieval system typically logs when it claims a match: one number.

Pair 1: cos = 0.78
 the query's nearest neighbor scores 0.78; the 2nd scores 0.44
 the region is sparse
 → the winner stands alone

Pair 2: cos = 0.78
 the nearest neighbor scores 0.78; the 2nd, 3rd, 4th score 0.77, 0.76, 0.76
 the region is crowded, several near-ties within a few hundredths of the winner
 → a coin toss dressed as a match

The scalar is identical. The situations, as constructed, are not: in Pair 1 the winner has no close competition; in Pair 2 three runners-up sit within 0.01–0.02 score points of the winner and the local neighborhood is dense with candidates that could plausibly have won instead. Whatever separates a decisive match from a fragile one in this sketch was available at retrieval time. Cosine, by construction, cannot express it — not because cosine is broken, but because it was never asked the question.

Change the Model, Change the Universe

Mon, 07 Sep 2026 13:00:00 +0000

Part V — Embedding Spaces Are Not Universal

The same sentence, three universes

Embed one sentence with three models:

model A (384-d): [ ... ]
model B (768-d): [ ... ]
model C (768-d): [ ... ]

A and B differ in length, so nobody expects to compare them coordinate-wise. But B and C are both 768-dimensional. It is tempting to line up their vectors and read something off the alignment directly — to compute cos(B_sentence, C_sentence) and interpret whatever number comes back.

Can One Embedding Space Be Translated Into Another?

Mon, 07 Sep 2026 14:00:00 +0000

Part VI — Crossing Embedding Spaces · Can a map be learned at all?

The question, stated carefully

We have two encoders, E_A and E_B, and a set of objects x. Each object has two representations: E_A(x) in space A, E_B(x) in space B. Chapter 16 established that these are distinct declared spaces with no coordinate correspondence assumed by default. When dimensions permit, raw cross-space arithmetic is numerically possible, but without an alignment contract it has no authorized cross-space semantics. That is not the same claim as “the spaces are unrelated.” Chapter 16 also measured, between several genuinely independent encoders, substantial neighborhood overlap and high linear CKA — real structural agreement, existing side by side with the absence of any authorized coordinate correspondence. Those two facts coexisting is exactly what makes this chapter’s question worth asking rather than settled in advance.

Retrieval Is Not Geometry

Wed, 09 Sep 2026 01:50:00 +0000

Part VI — Crossing Embedding Spaces · Did it find the counterpart, or recreate the neighborhood?

The result that looks like success

Fit a bridge, translate a held-out source vector, and ask the simplest possible question: does the exact native target counterpart show up nearby? On the Wave 6 benchmark, for a paired ridge bridge translating mxbai-embed-large-v1 into Qwen3-Embedding-8B, the answer is about as good as that question can get. Across 589 held-out sentences, using the same fitted bridge and the same translated vectors throughout:

What Should a Translation Preserve?

Wed, 09 Sep 2026 02:00:00 +0000

Part VI — Crossing Embedding Spaces · Whose geometry counts as success?

“Preserve the geometry” is not a complete instruction

A bridge translates vectors from space A into space B. The natural instinct is to say:

Preserve the source geometry while you translate.

That sounds obviously correct. If two source points are close, keep them close. If two source points are far apart, keep them far apart. Preserve pairwise cosine, distances, neighborhoods.

Can a Smaller Representation Preserve a Larger One?

Mon, 07 Sep 2026 15:00:00 +0000

Part VII — What Survives Transformation

Two vectors for one document

full_doc → E(full_doc) one vector, dimension d
summary(doc) → E(summary(doc)) one vector, dimension d

Both vectors have exactly the same dimension. What became smaller is the text — the number of words available to encode — not the vector. One embedding was produced from a full document; the other from a much shorter compression of it, passed through the identical embedding pipeline. This is not the dimensionality reduction Chapter 7 covered, and it is not the cross-space translation Part VI built — both the full document and its compression are embedded natively, by the same encoder, into the same space. The question is:

From Deltas to Operators

Mon, 07 Sep 2026 15:10:00 +0000

Part VII — What Survives Transformation

The other thing a subtraction might mean

king − man + woman ≈ queen is the famous demonstration that a direction in embedding space can correspond to a semantic relation. It is also, as Chapter 2 noted, partly curated and works best locally — and it has a long list of documented problems: the offset method’s success is entangled with plain cosine-neighborhood structure, so a “the direction transfers” result has to beat the baseline of ignoring the offset and returning the nearest neighbour of the source word (Linzen, 2016); and performance varies wildly by relation type (Rogers, Drozd & Li, 2017).

Building an Embedding Runtime

Mon, 07 Sep 2026 16:00:00 +0000

Part VIII — Embeddings Become Infrastructure

Why an embedding runtime became necessary

An embedding system used to look like this:

text → vector → cosine → top-k

Twenty-five chapters ago that was a reasonable sketch. It no longer is, because along the way this book kept asking questions that sketch has no place to answer:

Which exact pipeline — model, revision, normalization, prefix, truncation — produced this vector? (Chapters 1, 17)
Can two spaces even be compared, and does comparing them license mixing their vectors? (Chapters 16, 20)
Does a fitted map between two spaces preserve retrieval, or fine relations, or a calibrated threshold — and are those three different questions? (Chapters 18–21)
Does a compressed or whitened representation still support the task that mattered, or only the geometry that was easy to check? (Chapters 7, 8, 24)
Does a semantic edit admit a reusable vector operator at all, and how would we know before trusting one? (Chapter 25)
What happens after a chain of these — a translation, then a compression, then an edit? Does whatever passed the first hop still hold?

None of these questions has a script-sized answer. Each one produced a companion component — a space_record, a neighborhood_report, a calibration_record, a bridge, a preservation_profile, a compression_record, a transformation_record — because each is a measurement, and a measurement needs somewhere to live that is not the next paragraph of prose. Once there are seven or eight of these, kept as separate scripts, a system built from them accumulates a specific failure mode: nothing enforces that a translated vector still carries its bridge’s scope, or that a compressed one still carries its retention knee, or that anyone checks before mixing two spaces that were never compared. The instruments exist. Nothing makes using them the path of least resistance.

Preference Rankers — Learning Which Answer Is Better

Sat, 22 Aug 2026 01:18:00 +0100

Preference Rankers — Learning Which Answer Is Better

So far in Models From First Principles, we have mostly trained models by telling them what the answer should be.

MR.Q took a context and a response and produced a number:

context + response
 ↓
 model
 ↓
 0.82

That number might mean quality.

Or usefulness.

Or reward.

But there is an awkward question hiding underneath that architecture:

Where did the 0.82 come from?

MR.Q — Building a Neural Quality Model From Two Embeddings

Sat, 08 Aug 2026 14:33:00 +0100

MR.Q — Building a Neural Quality Model From Two Embeddings

In the previous post, we established the core idea behind this series:

A complicated model becomes understandable when you recursively decompose it into smaller models, blocks, layers and tensor operations.

Now we build the first real model.

Not a transformer.

Not a giant language model.

Not an agent.

A scorer.

We will take two embeddings:

context embedding
response embedding

combine them, encode the relationship between them, and predict one scalar:

RELATE: Searching Embeddings by Relation, Not Just Similarity

Wed, 05 Aug 2026 23:24:45 +0100

Embeddings are everywhere in modern AI.

They power semantic search, retrieval-augmented generation, recommendations, clustering, duplicate detection, code search, memory systems, and many of the mechanisms through which an AI system decides what information is relevant.

Yet most systems interrogate embeddings in essentially the same way:

Take two vectors and calculate cosine similarity.

That is useful. But it also makes a strong assumption.

It assumes that the information we care about is expressed directly through the default geometry of the embedding space.