Text Analysis

You do not throw whole, unwashed vegetables into a pot. You rinse them, peel them, and chop them into pieces of a consistent size first, because that is what makes them cook evenly. Text analysis does the same to raw text before it is indexed: it takes a messy sentence and turns it into clean, consistent pieces a search engine can actually work with.

In How Search Works the inverted index mapped a word to the documents that contain it. But it does not store the raw word. Between the document and the index sits an analyzer, a small pipeline that decides what a "term" even is. That pipeline quietly sets the ceiling on everything search can match, and most search bugs are really analysis bugs.

Tokenization

First the text is broken into tokens. "The cats are running." becomes The, cats, are, running. That sounds trivial until you meet punctuation, hyphens, URLs, C++, or languages that do not put spaces between words. The tokenizer is where those decisions get made, and a bad one can split e-mail into e and mail or lose C# entirely.

Normalization

Next the tokens are made uniform. Lowercasing is the obvious one: without it, Apple and apple are two different terms and a search for one misses the other. Analyzers also fold accents (café to cafe), strip diacritics, and normalize width and punctuation, so that trivial surface differences do not become missed matches.

Stopwords

Some words carry almost no signal. the, a, is, to, of appear in nearly every document, so matching them tells you nothing and they cost space in every postings list. Classic search drops these stopwords entirely. Modern engines are often more careful (removing the from "The Who" or "to be or not to be" destroys the query), but the idea, spend the index on words that discriminate, is still everywhere.

Stemming and lemmatization

Here is the one that matters most for recall. A user who searches run expects to find documents that say running, runs, or ran. Left alone, those are four different terms that never meet. Stemming chops words down to a rough root with fast rules: running to run, cats to cat, quickly to quick. It is crude and sometimes over-reaches (universe and university can collapse together), but it is cheap and it works. Lemmatization is the careful cousin: it uses a dictionary and part-of-speech to find the real base word, turning better into good and ran into run, at more cost. Either way, the point is to make different forms of the same idea land on the same term.

Synonyms

The last common step teaches the index that different words mean the same thing. A synonym filter expands or rewrites tokens so that NYC also matches New York, and heart attack also matches myocardial infarction. It is how a search stops depending on the user guessing the exact word your documents happened to use.

The one rule that breaks everything

Analysis runs in two places, and they must agree. The same analyzer that processed the document at index time has to process the query at search time. Index Running as run but search for the raw word running, and you get nothing, because running is not a key in the index; run is. Almost every "why does my search return zero results" bug is a mismatch between the index-time and query-time analyzers.

Watch a sentence go through the whole pipeline and turn into the terms that actually get stored:

Analysis is unglamorous and it decides everything. With clean terms in the index, the next question is how to rank the documents that match them, which is where BM25 comes in.