Queries are full of numbers - ages, ids, totals. Today you teach the tokenizer to read a run of digits into a single number token carrying its integer value.
Recognize a run of digits as one number token and attach its integer value.
A number literal is a run of digit characters that stands for an integer
value. The tokenizer reads consecutive digits, stops at the first non-digit, and
converts the collected text into an actual number stored on the token - so later
stages get 42 the integer, not "42" the string.
Keeping the parsed value on the token is the point: the parser and executor should never re-parse text. For now integers are all you need - decimals and negatives can wait, and negatives will fall out naturally later as an operator applied to a positive literal.
// read digits while unicode.IsDigit(c)// convert the collected text to an int64 for the token's valuetok := Token{Kind: TokNumber, Int: parsed}