Today the index starts analyzing every document you add and can hand back the terms it derived. This is the join between the store from lesson 1 and the analyzer you just finished - and the first point where the whole pipeline runs end to end.
Have the index analyze each document on add and expose its terms by id.
Analysis has lived in free functions so far. Today it moves into the index: when a
document is added, the index runs analyze on it once and stashes the resulting
terms alongside the original text. Analyzing eagerly, on add, means the work
happens a single time per document rather than on every search.
You now have a walking skeleton of the whole first act - add a document, and the index holds both what you can show a user (the original text) and what you can search (the terms). Try adding two or three documents and printing their terms; seeing the pipeline run on real input is the reward for the last week.
def add(self, doc_id, text):self._docs[doc_id] = textself._terms[doc_id] = analyze(text) # analyze once, on adddef terms(self, doc_id):return self._terms[doc_id]
Manning, Introduction to Information Retrieval - ch. 1.2.