Lesson 07 · Recording positions - Build a Search Engine

Recording positions

Today you extend analysis to remember where each term sat in the original token stream. Those positions are what will later let you answer phrase queries like "quick brown".

The goal

Analyze text into (term, position) pairs, where position is the term's index in the original token stream.

Start here - the target

TO DO

Scenario: Tracking each term's position

Giventhe text "the quick brown fox"

Whenit is analyzed with positions

Thenthe pairs are [("quick", 1), ("brown", 2), ("fox", 3)]

Andthe dropped stop word "the" leaves a gap - no term has position 0

Background

A phrase query like "quick brown" needs more than “both words appear” - it needs them adjacent. To know that later, you must remember where each term sat. Position is the token’s index in the original stream, assigned before stop words are dropped.

Keeping the original index is the subtle part. If you numbered terms only after removing stop words, then the quick and quick would look identical, and a phrase search could match words that were never actually next to each other. By preserving gaps where stop words used to be, adjacency stays honest.

Make it work

def analyze_positions(text):
    pairs = []
    for i, tok in enumerate(fold(tokenize(text))):
        # keep the ORIGINAL index i even when tokens are dropped
        if tok not in STOP_WORDS:
            pairs.append((stem(tok), i))
    return pairs