Today you wire the four steps you built - tokenize, fold, remove stop words, stem - into a single analyze() function. This one function is what turns any text into the terms your index will store.
Compose tokenizing, folding, stop-word removal, and stemming into one analyze() call.
Each of the last four lessons built one link; today you connect them into a chain. The order is not arbitrary: you tokenize first (nothing works on a raw string), fold before comparing against the lowercase stop-word set, and stem last so that suffixes come off canonical, lowercased words.
This analyze function is the single most important seam in the whole engine. The
exact same function must run on documents when you index them and on queries
when you search - if the two ever diverge, a query term and its matching document
term will no longer be spelled the same, and the match silently disappears.
def analyze(text):tokens = tokenize(text)tokens = fold(tokens)tokens = remove_stopwords(tokens)return [stem(t) for t in tokens]
Manning, Introduction to Information Retrieval - ch. 2.2.