Lesson 25 · Document length and average

Document length and average

Today you measure each document's length in tokens and the average length across the index. BM25 needs both to judge whether a document is long or short relative to its peers.

The goal

Report a document's token length and the average document length over the index.

Start here - the target

TO DO

Scenario: Measuring document lengths

Givenan index with ("d1", "cat dog") and ("d2", "cat dog cat dog") added

Whenyou measure lengths

Thendoc_len("d1") is 2 and doc_len("d2") is 4

Andthe average document length is 3.0

Background

Cosine used a vector length built from weights. BM25 uses a plainer notion: document length as the number of tokens, and the average document length across the whole index. cat dog has length 2; cat dog cat dog has length 4; the average over the two is 3.0.

The ratio of a document’s length to the average is what BM25 will use to decide whether a document is unusually long - and should have its term frequencies discounted - or unusually short. Count the analyzed terms you already stored per document, including repeats, since a repeated word genuinely adds to length. This is the last ingredient before the modern ranking function.

Make it work

def doc_len(self, doc_id):
    return len(self._terms[doc_id])       # analyzed term count
def avg_doc_len(self):
    lengths = [self.doc_len(d) for d in self._docs]
    return sum(lengths) / len(lengths)