Vector search sounds like machine-learning machinery. Underneath it are two plain ideas, and once they click, the build in the next post is short.
The two ideas are embeddings, which turn text into points positioned by meaning, and chunking, which cuts documents into pieces small enough to embed well.
An embedding is text as a point
An embedding model takes a piece of text and returns an array of numbers, a few hundred to a few thousand of them. That array is a point in a high-dimensional space.
The one property that matters: the model places text by meaning. Sentences that mean similar things land near each other; sentences that mean different things land far apart. "Cat" and "kitten" sit close. "Cat" and "car", almost the same spelling, sit far apart, because the model was trained on meaning, not letters.
def get_embeddings(texts: list[str]) -> np.ndarray:
"""Turn each text into an embedding vector."""
response = client.embeddings.create(input=texts, model=EMBEDDING_MODEL)
return np.array([row.embedding for row in response.data])
One API call in, an array of points out. That is the whole interface.
Closeness is measured by angle
If text is a point, "similar" means "close". The usual way to measure closeness between two embeddings is cosine similarity: the cosine of the angle between the two vectors. It runs from 1 (pointing the same way, very similar) down toward 0 (unrelated).
from sklearn.metrics.pairwise import cosine_similarity
texts = ["A cat sat on the mat.",
"A kitten played with a toy.",
"A dog ran across the park."]
emb = get_embeddings(texts)
cosine_similarity([emb[0]], [emb[1]]) # cat vs kitten -> high
cosine_similarity([emb[0]], [emb[2]]) # cat vs dog -> lower
The cat sentence scores closer to the kitten sentence than to the dog sentence, even though none of them share the important words. Keyword search would have called all three unrelated. The embedding sees that cat and kitten belong together.
Text becomes points positioned by meaning. Cosine similarity measures the angle between them.
Modern embedding models even read context. The word "bank" lands in a different place when the sentence is about a river than when it is about money, so the same word gets two positions depending on what it means in that sentence.
Chunking: why you cannot embed a whole document
Embedding a thousand-page manual as a single vector fails, for two reasons.
The document might exceed the model's input limit, so it will not embed at all. And even if it fits, one vector for the whole thing is a blur: it averages every topic in the document into a single point, so a search for any specific topic matches it weakly.
The fix is to split the document into small pieces, called chunks, and embed each chunk separately. A search then returns the chunk that is actually relevant, not the whole document.
Chunk size is a real tradeoff
How big a chunk should be is a genuine decision, with a failure at each extreme.
Too small, and a chunk loses its context. A chunk that reads only "It supports up to 20 concurrent connections" cannot tell you what "it" is. The meaning leaked out at the cut.
Too large, and a chunk blurs. Several topics share one vector, so the chunk matches many queries weakly and none of them well, which is the whole-document problem back at a smaller scale.
A common range is 200 to 1,000 tokens, roughly 150 to 750 English words, and the right size depends on the material. Dense reference text wants smaller chunks; flowing narrative wants larger ones to keep a thought intact.
Overlap keeps sentences whole
Cutting every 500 characters has an obvious hazard: the cut can land in the middle of a sentence, and the two halves end up in different chunks, each meaningless.
Overlap fixes it. Let each chunk share a slice of text with the one before, so a sentence split at a boundary survives whole in at least one chunk.
def chunk(text: str, size: int = 500, overlap: int = 50) -> list[str]:
"""Split text into overlapping chunks so boundaries do not sever meaning."""
chunks, start = [], 0
while start < len(text):
chunks.append(text[start:start + size])
start += size - overlap # step back by the overlap each time
return chunks
The size - overlap step is the whole trick: each chunk begins a little before the last one ended, so
no boundary cleanly cuts a thought in two.
Too small loses context, too large blurs topics. Overlap keeps a boundary sentence whole.
What to take from this
- An embedding turns text into a point positioned by meaning, so similar text lands close and cosine similarity measures how close.
- A whole document embeds badly, so split it into chunks and embed each one, returning the relevant chunk instead of the whole file.
- Chunk size is a tradeoff: too small loses context, too large blurs topics, and overlap stops a cut from severing a sentence.
Points and chunks are the raw materials. The next post assembles them into working vector search, and shows why a real system needs a vector database rather than a loop over an array.