Today you split raw text into word tokens by breaking on everything that is not a letter or digit. This is the first step of turning a document into something searchable.
Split a string into a list of word tokens, discarding punctuation and spacing.
A document is one long string, but search works on words. Tokenizing is the act of chopping that string into individual tokens. The simplest useful rule: a token is a maximal run of letters and digits, and everything else - spaces, commas, exclamation marks - is just a separator that gets thrown away.
This rule is crude on purpose. It splits don't into don and t, and it has no
idea about hyphenated words. That is fine: a simple, predictable tokenizer is easy
to reason about, and every later normalization step assumes it runs first.
# a token is a maximal run of letters/digits;# everything else is just a separatorimport redef tokenize(text):... # find every run of [A-Za-z0-9]
Manning, Introduction to Information Retrieval - ch. 2.2.