Projects/Build a Search Engine

Build a Search Engine

Build a small search index in the spirit of Lucene or Elasticsearch: add documents, then search them. Each lesson opens with a concrete spec and closes with it satisfied, until the analyzer, index, and ranking all hold together.

34 lessonsSmall~20 min / lessonInverted indexTF-IDFBM25

The project

What you'll build over the next 34 lessons

Over the next 34 lessons you build a working in-memory search index library from scratch - a small core in the spirit of Lucene or Elasticsearch. You add documents to it and search them: an analyzer turns text into normalized terms, an inverted index maps every term to the documents and positions where it appears, and a ranking layer scores matches with TF-IDF, cosine normalization, and BM25.

You finish with a coherent library you import and use: construct an index, add documents, and run BM25-ranked free-text search alongside boolean and phrase queries, with highlighted snippets for results. It is a teaching-grade core - clear and correct, held in memory rather than persisted, sharded, or compressed like a production system - and every lesson ends green with the public API importable and working.

build-a-search-engine / lesson-01.md

Lesson 01 · Documents & analysis

The index and its documents

Today you create the core object of the whole project - a search index you can add documents to and read them back. Everything else you build hangs off this one type.

The goal

Build a SearchIndex that stores documents by id and reports how many it holds.

Start here - the target

TO DO

Scenario: Adding and retrieving documents

Givena new empty search index

Whenyou add ("d1", "hello world") and ("d2", "goodbye")

Thenindex.size is 2

Andindex.document("d1") is "hello world"

Background

Every search engine begins with the same humble object: a place to put documents. Before you can tokenize, index, or rank anything, you need a document store - a mapping from a document id to its original text. Keeping the raw text around matters: later you will show snippets of it in results, so the store is not just a stepping stone.

Keep it tiny. A dictionary from id to text, an add method, a size, and a document(id) lookup. This is the spine the rest of the project attaches to - every later lesson adds a method to this same class.

Make it work

class SearchIndex:
    def __init__(self):
        self._docs = {}          # id -> original text
    def add(self, doc_id, text):
        ...                      # store it
    # size, document(id) read it back

Where this project stops - and where to go next

The library implements BM25 free-text ranking, boolean AND/OR/NOT, phrase queries, and snippets end to end with fail-fast error handling, but the boolean grammar has no operator precedence and only free-text search is BM25-ranked.

Extend it next

Add operator precedence and parentheses to boolean queries (currently a strict left-to-right fold)
Rank phrase and boolean query results with BM25 instead of returning bare unordered doc-id lists
Replace the four-rule suffix stripper with a real stemmer (Porter/Snowball)
Highlight every matched occurrence in a snippet, not just the first
Add index persistence (save/load to disk)
Support named fields per document (e.g. title vs. body) with per-field weights

Books & references that go deeper

Introduction to Information Retrieval · Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze
The standard textbook on indexing, ranking, and evaluation - free online from the authors.
Managing Gigabytes · Ian H. Witten, Alistair Moffat, Timothy C. Bell
The classic reference on index compression and large-scale text retrieval.
A Vector Space Model for Automatic Indexing · Gerard Salton, A. Wong, C. S. Yang
The 1975 paper that introduced the vector space model underlying TF-IDF ranking.
Search Engines: Information Retrieval in Practice · W. Bruce Croft, Donald Metzler, Trevor Strohman
A more implementation-minded companion, with worked examples of indexing and ranking pipelines.