Parsing happens in two stages, and the first is tokenizing (also called lexing): turning the raw formula string into a flat list of meaningful atoms. The parser never looks at characters - it reads tokens like “the number 12”, “a plus”, “an open paren”. A key detail is that a number is one token even when it spans several digits, so 12 scans as a single Number 12, not two separate digits.

Two setup decisions pay off later. First, a formula always begins with =, so the tokenizer strips that leading sign before scanning - by the time the parser runs, the = is gone. Second, define the entire token set now, including kinds we cannot produce yet (cell references, colons, commas, comparison operators). Naming them all up front means the parser’s later lessons only add scanning rules, never reshape the token type. End every stream with an explicit EOF token so the parser always knows when input has run out.