Embeddings From First Principles on AIBussin — AI applications, systems and books

What Is an Embedding?

Mon, 07 Sep 2026 09:00:00 +0000

Part I — A Vector Is Not Meaning

Three words and a list of numbers

Take three words:

cat
dog
airplane

Turn them into numbers. Any embedding API will do it. You get three arrays, each maybe 384 or 768 or 1,536 floats long:

cat [ 0.021, -0.114, 0.062, ... ]
dog [ 0.019, -0.098, 0.071, ... ]
airplane [-0.087, 0.203, -0.041, ... ]

Compute the angle between cat and dog. It is small. Compute the angle between cat and airplane. It is larger. Something about “cats and dogs are both pets” appears to have survived the trip into number-space.

Meaning Becomes Geometry

Mon, 07 Sep 2026 09:10:00 +0000

Part I — A Vector Is Not Meaning

A space you can draw

Give six words two coordinates each, by hand:

 x = monarchy / institutional power y = gendered association (-1 … +1)
king ( 0.9, 0.6)
queen ( 0.9, -0.6)
duke ( 0.7, 0.5)
duchess ( 0.7, -0.5)
apple (-0.7, 0.1)
orange (-0.7, -0.1)

Now several semantic questions have a geometric form:

Which words form a group? → apple and orange sit almost on top of each other, about 0.2 apart, and far from the four titles. The titles form a looser group of their own near x ≈ 0.8. Grouping shows up as density.
What distinguishes king from queen? → the vector king − queen ≈ (0, 1.2). It points almost purely along y. The difference between them, in this space, is one attribute. (duke − duchess ≈ (0, 1.0) points the same way, but only because we placed the points that way; a learned space owes us no such consistency.)
Is apple oriented like king? → the angle between them is obtuse and their cosine is negative. Fruit and titles point into different half-planes. Orientation separates the two categories cleanly, even though, as we are about to see, it is noisier within a category.
Are king and queen related? → obviously: they are the two monarchs. But their Euclidean distance is 1.2, while king sits 0.22 from duke and about 1.12 from duchess. Rank king’s neighbors by distance and queen comes third, behind duke and duchess. The gender axis — which a search for “royal titles” need not care about at all — has quietly reordered the neighborhood.

That last point is the one to carry forward.

Learning an Embedding Space

Mon, 07 Sep 2026 09:20:00 +0000

Part I — A Vector Is Not Meaning

Refusing the handout

For two chapters the vectors were given. Chapter 1 took them from an API and asked what was in them; Chapter 2 placed some by hand and treated the rest as the output of a model we agreed not to open. Chapter 2 closed on the question it left unanswered: where does the geometry come from?

This chapter builds a small space from nothing but a corpus and a counting rule. A usable geometry appears — related words land near each other without anyone labeling an axis “animal” or “drink.” Then comes the more important half: we inspect the construction closely enough to predict, from the mechanism alone, what it cannot preserve. When the measured results arrive, the failures are not surprises. They are consequences of the path the information took.

Similarity Is a Decision

Mon, 07 Sep 2026 09:30:00 +0000

Part I — A Vector Is Not Meaning

Two vectors, four answers

Here are three document vectors, kept to three dimensions so every number is checkable:

x = ( 2.0, 0.0, 0.0 ) a short doc, one strong topic
y = ( 6.0, 0.1, 0.0 ) a long doc, same topic, much more of it
z = ( 0.0, 2.0, 0.0 ) a short doc, a different topic

Before computing anything, look at the geometry. x and y point in almost exactly the same direction but have very different lengths (norm 2 versus about 6). x and z have the same length but point ninety degrees apart. So “is x more like y or z?” comes down to a prior question: which kind of difference should count — a difference in direction, or a difference in size?

Dimensions Do Not Mean What You Think

Mon, 07 Sep 2026 10:00:00 +0000

Part II — Inside the Space

What does dimension 173 mean?

Pull the 173rd coordinate of every vector in your corpus and sort. You get a list of documents ordered by something. Once in a while a coordinate is weakly readable — “this one runs high for questions” — but usually the sorted list has no nameable theme. Dimension 173 is not “formality,” not “sentiment,” not “is about sports.”

And yet the vectors work. Similarity search returns sensible results, clusters fall out, nearest neighbors are reasonable. The usable structure is there. It is just not sitting on the individual axes.

Neighborhoods and Hubs

Mon, 07 Sep 2026 10:10:00 +0000

Part II — Inside the Space

The question a vector cannot answer alone

Hand someone a single embedding vector — [0.13, -0.82, ...] — and ask what it means. They cannot say. Chapter 5 is why: the basis is not identifiable, the scale is model-specific, no coordinate names a concept.

Hand them the same vector and its ten nearest neighbors — all customer-support apologies, all in the same register — and they can usually name the topic, the register, and roughly what the point is about.

How Many Dimensions Does Meaning Need?

Mon, 07 Sep 2026 10:20:00 +0000

Part II — Inside the Space

A representation with several sizes

all-mpnet-base-v2 emits vectors of length 768. Embed the 1,173 RELATE items with it and ask several different questions about the size of the resulting cloud, and you do not get one answer:

nominal dimension 768
centered matrix-rank ceiling 768 (= min(1,173−1, 768))
95%-of-variance dimension 205
participation ratio 67.0
entropy effective rank 387.4
intrinsic-dimension estimate (MLE, k=10) 6.81
intrinsic-dimension estimate (TwoNN) 4.23

The Wave 2 dimensionality artifact records the last five numerical summaries plus nominal dimension; it does not persist an observed matrix-rank value, so the line above reports only the algebraic ceiling. Same vectors, same corpus: the reported or implied notions of size still range from about 4 to 768 — almost two orders of magnitude.

The Shape of an Embedding Space

Mon, 07 Sep 2026 10:30:00 +0000

Part II — Inside the Space

Same corpus, five geometries

Freeze the input. Embed exactly the same 1,173 RELATE items with five models and, before scoring any task, measure the geometry each one imposes on identical text.

The evaluation harness requests L2-normalized embeddings (normalize_embeddings=True), so every vector analyzed in this chapter lies on the unit sphere, and cosine is purely a statement about direction.

MEASURED on RELATE v0.1, Wave 2 row 2.8 — artifact experiments/embeddings-from-first-principles/wave2/artifacts/shape-comparison.json. All descriptors computed from one run on the frozen corpus.

From Similarity to Search

Mon, 07 Sep 2026 11:00:00 +0000

Part III — Retrieval Is an Experiment

Retrieval in four lines

# assume embed() returns L2-normalized vectors, so dot product == cosine (Chapter 4)
query_vec = embed(query) # one unit vector
scores = corpus_vecs @ query_vec # a cosine score for every document
order = np.argsort(-scores) # rank all n documents, best first
results = [corpus[i] for i in order[:k]] # keep the top k

That is the whole primitive for the simple policy shown here. A vector database may add several different kinds of machinery around it: an ANN index or quantization may approximate the same ranking more cheaply; sharding may distribute the computation; filtering may change the eligible candidate set; persistence and replication may make the system operable. Do not collapse all of those into “faster search” — some preserve the target policy, some approximate it, and some redefine it.

The Nearest Neighbor Can Be Wrong

Mon, 07 Sep 2026 11:10:00 +0000

Part III — Retrieval Is an Experiment

The result that is closest and wrong

Illustrative example — not a RELATE measurement.

Query:

Is Dublin the capital of Ireland?

Imagine the passage that ranks first is this one:

Dublin is not, and has never been, the capital of Ireland — that
distinction belongs to the older seat of government at Tara.

Fluent, on-topic, confidently phrased, and among the geometrically closest things in the corpus — and false. A model handed this passage as context may repeat its claim. The retrieval system did its job: it found a near vector. “Near” was not “correct.”

Hard Negatives

Mon, 07 Sep 2026 11:20:00 +0000

Part III — Retrieval Is an Experiment

Same model, four negative sets, four verdicts

MEASURED on RELATE v0.1, Wave 1 row 1.8 — artifact experiments/embeddings-from-first-principles/wave1/artifacts/margin-collapse.json. Model all-mpnet-base-v2, unchanged across every row below.

negative-selection rule mean margin selected-positive win rate
same-domain random, up to 5 per query 0.4738 1.0000
in-model selector, as implemented 0.2094 0.9814
structured perturbations 0.0932 0.9331
BM25 lexical top-5 0.0595 0.7621

The model did not change. The 269-query set did not change. The reference-positive rule did not change. Only the rule that picked which passages the positive had to beat changed — and the mean margin ranged from 0.4738 to 0.0595, a nearly eightfold difference between the widest and narrowest measured conditions.

Retrieval Is a Policy

Mon, 07 Sep 2026 11:30:00 +0000

Part III — Retrieval Is an Experiment

“Just retrieve the relevant documents”

There is no such operation. Chapter 11 closed on a single insight: a score never travels alone — it arrives with a negative-selection rule that is part of the measurement. This chapter widens the frame. A retrieval result never travels alone either. It arrives wrapped in a query construction, a representation, a candidate-eligibility rule, a scoring and fusion policy, a selection rule, an execution strategy, and a context-assembly step that decides what a downstream model actually sees.

How Do You Evaluate an Embedding?

Mon, 07 Sep 2026 12:00:00 +0000

Part IV — Measuring the Representation

Same encoders, same item pool, different winner

Three encoders. One item pool. One typed-pair pool. One metric implementation. Three relevance definitions, applied consistently. The encoder models and item-side corpus are unchanged between the two releases; what expands is the query population, from 269 queries to 411.

RELATE v0.1 (269 queries): all-mpnet-base-v2 wins every relevance definition
RELATE v0.2 (411 queries): bge-large-en-v1.5 wins every relevance definition

That is not a hypothetical. It is Wave 1 row 1.9, run twice against two frozen releases, and it is this chapter’s demonstration in miniature before the demonstration itself. The models did not change. The evaluation workload did. The winner changed.

Calibration

Mon, 07 Sep 2026 12:10:00 +0000

Part IV — Measuring the Representation

What is 0.81?

Illustrative scenario — invented values for intuition, not a measured result.

A pipeline decides two documents are “duplicates” if their cosine exceeds 0.8. Someone picked 0.8 because it looked reasonable. Imagine the two populations that number is actually sitting between:

distribution of cosine for KNOWN duplicate pairs: mean ~0.79
distribution of cosine for KNOWN non-duplicate pairs: mean ~0.71

If those two populations overlap the way this sketch implies, the threshold sits inside both: a true duplicate at 0.78 is rejected, an unrelated pair at 0.82 is accepted. This chapter’s actual experiment, later, replaces the sketch with measured positive and negative score samples and derived discrimination/operating statistics. The artifact does not persist the full distributions themselves, and its 85.6% figure is the coverage of a specific quantile-derived band rather than a generic measure of distributional overlap.

Is Similarity One-Dimensional?

Mon, 07 Sep 2026 12:20:00 +0000

Part IV — Measuring the Representation

Two pairs, same cosine, different situations

Illustrative vignette — constructed for intuition, not a measured RELATE example.

Consider what a retrieval system typically logs when it claims a match: one number.

Pair 1: cos = 0.78
 the query's nearest neighbor scores 0.78; the 2nd scores 0.44
 the region is sparse
 → the winner stands alone

Pair 2: cos = 0.78
 the nearest neighbor scores 0.78; the 2nd, 3rd, 4th score 0.77, 0.76, 0.76
 the region is crowded, several near-ties within a few hundredths of the winner
 → a coin toss dressed as a match

The scalar is identical. The situations, as constructed, are not: in Pair 1 the winner has no close competition; in Pair 2 three runners-up sit within 0.01–0.02 score points of the winner and the local neighborhood is dense with candidates that could plausibly have won instead. Whatever separates a decisive match from a fragile one in this sketch was available at retrieval time. Cosine, by construction, cannot express it — not because cosine is broken, but because it was never asked the question.

Change the Model, Change the Universe

Mon, 07 Sep 2026 13:00:00 +0000

Part V — Embedding Spaces Are Not Universal

The same sentence, three universes

Embed one sentence with three models:

model A (384-d): [ ... ]
model B (768-d): [ ... ]
model C (768-d): [ ... ]

A and B differ in length, so nobody expects to compare them coordinate-wise. But B and C are both 768-dimensional. It is tempting to line up their vectors and read something off the alignment directly — to compute cos(B_sentence, C_sentence) and interpret whatever number comes back.

Versioning the Space

Mon, 07 Sep 2026 13:10:00 +0000

Part V — Embedding Spaces Are Not Universal

The upgrade that broke search quietly

Illustrative scenario — not a measured incident.

Production runs embedding model v1. A better v2 ships. Someone updates the client library. New documents get v2 vectors; the existing documents are still carrying v1 vectors. Nothing errors. Queries are embedded with v2 and scored directly against a mix of v1 and v2 document vectors.

No exception is raised, because none of this is mechanically invalid — a v2 query and a v1 document, at the same dimension, dot-product into a perfectly ordinary float. What actually breaks is quieter than an error: the index now contains vectors from two declared spaces while every query is produced in only one of them, so a portion of the comparisons the system runs no longer satisfy the representation contract either side assumed. Retrieval keeps returning ten plausible-looking results. Nothing about the response shape announces that some of those ten were scored by an unsupported cross-space comparison. That is exactly the danger: silent retrieval-contract violation, not silent catastrophe.

Can One Embedding Space Be Translated Into Another?

Mon, 07 Sep 2026 14:00:00 +0000

Part VI — Crossing Embedding Spaces · Can a map be learned at all?

The question, stated carefully

We have two encoders, E_A and E_B, and a set of objects x. Each object has two representations: E_A(x) in space A, E_B(x) in space B. Chapter 16 established that these are distinct declared spaces with no coordinate correspondence assumed by default. When dimensions permit, raw cross-space arithmetic is numerically possible, but without an alignment contract it has no authorized cross-space semantics. That is not the same claim as “the spaces are unrelated.” Chapter 16 also measured, between several genuinely independent encoders, substantial neighborhood overlap and high linear CKA — real structural agreement, existing side by side with the absence of any authorized coordinate correspondence. Those two facts coexisting is exactly what makes this chapter’s question worth asking rather than settled in advance.

Alignment

Mon, 07 Sep 2026 14:10:00 +0000

Part VI — Crossing Embedding Spaces · Which map family survives held-out data?

What Chapter 18 left open

Chapter 18 fit the simplest serious bridge — affine ridge regression — from all-MiniLM-L6-v2 into all-mpnet-base-v2, and evaluated it on entity families the map never saw. The result was not a verdict. It was a profile, uneven enough to make one number impossible to trust on its own:

coordinate reconstruction 0.5860
10-NN neighborhood overlap 0.7128
retrieval nDCG@10 ratio 0.7987
rank-triplet agreement 0.7529
calibration-transfer score 0.7882
relation-profile Pearson correlation 0.8673
structured hard-negative margin ratio 0.2342
held-out/train reconstruction ratio 0.6480

The chapter’s lesson was not “ridge works.” It was that ≈ is a preservation contract — a named property, measured on held-out data, not a single number a map either clears or fails. That leaves an open question this chapter exists to answer:

The Embedding Bridge

Mon, 07 Sep 2026 14:20:00 +0000

Part VI — Crossing Embedding Spaces · When may a system actually use one?

A map is not a bridge

Chapter 19 produced several fitted candidates — a Procrustes pipeline with its dimensionality adapter, an affine ridge map, a small MLP — each with its own held-out preservation profile, and none of them a declared winner. Chapter 19 ended deliberately unresolved: status: measured_not_authorized. Deploying any one of those transformations as “space A and space B are now compatible” is the mistake this chapter exists to prevent.

Did the Bridge Preserve the Space?

Mon, 07 Sep 2026 14:30:00 +0000

Part VI — Crossing Embedding Spaces · What did each measurement actually certify?

One bridge, eight measurements

Chapter 20 built the machinery that turns preservation evidence into a scoped authorization — a requirement, a measured value, and a decision, all cited together. What it deliberately did not do is ask how much any one of those measured values is actually entitled to say. That is this chapter’s entire job.

A preservation metric is a sensor. Before trusting its reading, ask which failure modes the sensor is even capable of seeing.

Retrieval Is Not Geometry

Wed, 09 Sep 2026 01:50:00 +0000

Part VI — Crossing Embedding Spaces · Did it find the counterpart, or recreate the neighborhood?

The result that looks like success

Fit a bridge, translate a held-out source vector, and ask the simplest possible question: does the exact native target counterpart show up nearby? On the Wave 6 benchmark, for a paired ridge bridge translating mxbai-embed-large-v1 into Qwen3-Embedding-8B, the answer is about as good as that question can get. Across 589 held-out sentences, using the same fitted bridge and the same translated vectors throughout:

What Should a Translation Preserve?

Wed, 09 Sep 2026 02:00:00 +0000

Part VI — Crossing Embedding Spaces · Whose geometry counts as success?

“Preserve the geometry” is not a complete instruction

A bridge translates vectors from space A into space B. The natural instinct is to say:

Preserve the source geometry while you translate.

That sounds obviously correct. If two source points are close, keep them close. If two source points are far apart, keep them far apart. Preserve pairwise cosine, distances, neighborhoods.

Can a Smaller Representation Preserve a Larger One?

Mon, 07 Sep 2026 15:00:00 +0000

Part VII — What Survives Transformation

Two vectors for one document

full_doc → E(full_doc) one vector, dimension d
summary(doc) → E(summary(doc)) one vector, dimension d

Both vectors have exactly the same dimension. What became smaller is the text — the number of words available to encode — not the vector. One embedding was produced from a full document; the other from a much shorter compression of it, passed through the identical embedding pipeline. This is not the dimensionality reduction Chapter 7 covered, and it is not the cross-space translation Part VI built — both the full document and its compression are embedded natively, by the same encoder, into the same space. The question is:

From Deltas to Operators

Mon, 07 Sep 2026 15:10:00 +0000

Part VII — What Survives Transformation

The other thing a subtraction might mean

king − man + woman ≈ queen is the famous demonstration that a direction in embedding space can correspond to a semantic relation. It is also, as Chapter 2 noted, partly curated and works best locally — and it has a long list of documented problems: the offset method’s success is entangled with plain cosine-neighborhood structure, so a “the direction transfers” result has to beat the baseline of ignoring the offset and returning the nearest neighbour of the source word (Linzen, 2016); and performance varies wildly by relation type (Rogers, Drozd & Li, 2017).

Building an Embedding Runtime

Mon, 07 Sep 2026 16:00:00 +0000

Part VIII — Embeddings Become Infrastructure

Why an embedding runtime became necessary

An embedding system used to look like this:

text → vector → cosine → top-k

Twenty-five chapters ago that was a reasonable sketch. It no longer is, because along the way this book kept asking questions that sketch has no place to answer:

Which exact pipeline — model, revision, normalization, prefix, truncation — produced this vector? (Chapters 1, 17)
Can two spaces even be compared, and does comparing them license mixing their vectors? (Chapters 16, 20)
Does a fitted map between two spaces preserve retrieval, or fine relations, or a calibrated threshold — and are those three different questions? (Chapters 18–21)
Does a compressed or whitened representation still support the task that mattered, or only the geometry that was easy to check? (Chapters 7, 8, 24)
Does a semantic edit admit a reusable vector operator at all, and how would we know before trusting one? (Chapter 25)
What happens after a chain of these — a translation, then a compression, then an edit? Does whatever passed the first hop still hold?

None of these questions has a script-sized answer. Each one produced a companion component — a space_record, a neighborhood_report, a calibration_record, a bridge, a preservation_profile, a compression_record, a transformation_record — because each is a measurement, and a measurement needs somewhere to live that is not the next paragraph of prose. Once there are seven or eight of these, kept as separate scripts, a system built from them accumulates a specific failure mode: nothing enforces that a translated vector still carries its bridge’s scope, or that a compressed one still carries its retention knee, or that anyone checks before mixing two spaces that were never compared. The instruments exist. Nothing makes using them the path of least resistance.