build-a-programming-language / lesson-04.md
Lesson 04 · Tokenizing

Integer literals

Numbers are the first token whose length you do not know up front - a digit could be followed by more digits. Today you teach the lexer to read a whole run of digits into a single INT token.

The goal

Read a run of digit characters into one INT token carrying the full number as its literal.

Start here - the target
TO DO
Scenario: Lexing multi-digit integers
Giventhe source string 12+3
Whenthe lexer produces tokens until the end
Thenthe tokens are INT "12", PLUS "+", INT "3", EOF
Andthe literal of the first token is the two-character string 12, not 1
Background

Single-character tokens were easy because you knew where they ended. A number is the first multi-character token: when the lexer sees a digit, it does not yet know how many more digits follow. So instead of returning immediately, it reads forward while the character is still a digit, then stops on the first non-digit.

The token’s literal is the whole slice of digits - "12", not 1 followed by a mystery. Keep the literal as text for now; the parser will convert it to an actual number later. Identifiers, in the next lesson, use exactly this read-while-it-matches shape.

Make it work
func (l *Lexer) readNumber() string {
start := l.position
for isDigit(l.ch) { l.readChar() }
return l.input[start:l.position]
}
func isDigit(ch byte) bool { return '0' <= ch && ch <= '9' }
CheckpointDONE
The lexer reads whole integers as one INT token. Commit.