Today you lowercase every token so that "The" and "the" become the same term. Without this, a search for one capitalization would miss all the others.
Fold a list of tokens to lowercase, leaving digits and already-lowercase tokens unchanged.
If Search, search, and SEARCH land in the index as three different terms, a
query for one will miss documents that used another. Case folding collapses
them by lowercasing every token, so capitalization stops mattering.
Digits and tokens that are already lowercase pass through untouched - lowercasing is idempotent, which is exactly what you want from a normalization step. This is a one-line transform today, but it is one link in the analysis pipeline you will assemble in a few lessons.
def fold(tokens):return [t.lower() for t in tokens]
Manning, Introduction to Information Retrieval - ch. 2.2.3.