Numbers are the atoms of every expression, so the tokenizer needs to recognize a run of digits as a single Number token. Today you scan one integer and capture both its text and where it started.
Scan a run of digit characters into a single Number token.
A number is more than one character, so scanning it means consuming a run: when
the cursor lands on a digit, keep advancing a second cursor while it still sees
digits, then emit one Number token spanning the whole run. The token’s text is the
exact slice of input it covered, and its position is where the run started, not
where it ended. Capturing the start position now is what will let a later error
message say “the number at position 0.”
The outer loop advances one character at a time, but a number consumes several at once, so after emitting the token you jump the main cursor past the whole run. Getting that jump right, so the cursor never rescans a character it already consumed, is the small discipline every scanner rule follows.
// walk a cursor; when it lands on a digit, consume the whole runfor i < len(in) {c := in[i]switch {case c >= '0' && c <= '9':j := ifor j < len(in) && in[j] >= '0' && in[j] <= '9' { j++ }toks = append(toks, Token{Number, in[i:j], i})i = j}}