build-a-search-engine / lesson-05.md
Lesson 05 · Documents & analysis

Stemming to a root

Today you strip common suffixes so that "jumping", "jumped", and "jumps" all reduce to one root term. This lets a query for one form match documents that used another.

The goal

Reduce a token to a root by stripping a common suffix.

Start here - the target
TO DO
Scenario: Stripping suffixes to a shared root
Giventhe words "jumping", "jumped", "cats", and "class"
Wheneach is stemmed
Then"jumping" and "jumped" both stem to "jump"
And"cats" stems to "cat", but "class" stays "class"
Background

Stemming trims inflectional endings so that related words meet at a common root. A full stemmer like Porter’s has dozens of rules; yours needs only a few to show the idea. Apply them longest-suffix first: strip -ing, then -ed, then a trailing plural -s.

The interesting cases are the guards. Only strip a suffix if enough word remains (so is does not become the empty string), and never strip the -s from a word ending in -ss like class, or you would mangle it. This crude stemmer will sometimes over-trim - that is a known trade-off, and consistency matters more than linguistic perfection, because the same rule runs on queries too.

Make it work
def stem(word):
# try suffixes longest-first; guard so short words survive
# -ing / -ed / plural -s ; but never strip -ss
...
Further Reading

M. F. Porter, "An algorithm for suffix stripping" (1980).

CheckpointDONE
Word variants collapse to a shared root, so different inflections match. Commit and stop here.