Lesson 26 · BM25 term saturation - Build a Search Engine

BM25 term saturation

Today you build BM25's term-frequency component, which saturates - the fifth occurrence of a word adds far less than the first, approaching a ceiling instead of growing forever. It is a sharper answer to repetition than the logarithm.

The goal

Compute BM25's saturating term-frequency factor, tf*(k1+1)/(tf+k1) with k1 = 1.5.

Start here - the target

TO DO

Scenario: Diminishing returns on repetition

Giventhe BM25 term factor with k1 = 1.5

Whenit is applied to term frequencies

Thena tf of 1 gives 1.0

Anda tf of 3 gives 1.6667 (rounded to 4 places)

Background

BM25 is the ranking function most modern search engines reach for, and its handling of term frequency is the heart of it. Instead of growing without bound, its term factor saturates: tf * (k1 + 1) / (tf + k1). As tf climbs, the value rises quickly then flattens, approaching a ceiling of k1 + 1. A single occurrence already earns 1.0; three earn only about 1.67.

The parameter k1 (commonly 1.2 to 2.0; use 1.5) tunes how fast it saturates

smaller k1 flattens sooner. This is a more principled curve than the logarithm: it directly encodes “the first mention matters most, and beyond a point more mentions barely move the needle.” Tomorrow you fold in document length to complete the formula.

Make it work

K1 = 1.5
def bm25_tf(tf):
    # rises fast at first, then flattens toward k1 + 1
    return tf * (K1 + 1) / (tf + K1)