Not every number is an integer, so the scanner needs to accept a decimal point inside a number run. Today you extend number scanning to allow a dot, then confirm the tokenizer handles a full expression end to end.
Extend the number rule so a decimal point is part of the number run.
A decimal like 3.14 is still one number, so the fix is small: let the dot both
begin a number run and continue one, alongside the digits. The run 3.14 is scanned
as a single Number token whose text is "3.14", and the operator that follows it
picks up at its own position. This is deliberately permissive; it will happily scan
3.1.4 too, and leaving that to be caught later (when the text is converted to an
actual number) keeps the scanner simple and its rules independent.
With this, the tokenizer is complete for arithmetic: it turns "3.14 + 2" into the
clean stream Number "3.14", Operator "+", Number "2", EOF. That stream, with
its kinds, texts, and positions, is the entire interface between the scanner and the
parser you start building next. Everything from here reads tokens, never characters.
// include '.' both to start a number and to continue onecase (c >= '0' && c <= '9') || c == '.':j := ifor j < len(in) && ((in[j] >= '0' && in[j] <= '9') || in[j] == '.') { j++ }toks = append(toks, Token{Number, in[i:j], i}); i = j