Lesson 08 · Tokenizing a whole program - Build a Programming Language

Tokenizing a whole program

Your lexer now knows every token kind the language uses. Today you point it at a realistic line of source and wire it into the REPL, so typing code prints the exact token stream - the first end-to-end run of the pipeline.

The goal

Lex a full statement into its complete token stream and have the REPL print those tokens.

Start here - the target

TO DO

Scenario: Lexing a realistic statement

Giventhe source line: let add = fn(x, y) { x + y };

Whenthe lexer produces tokens until the end

Thenthe token types are LET, IDENT, ASSIGN, FUNCTION, LPAREN, IDENT, COMMA, IDENT, RPAREN, LBRACE, IDENT, PLUS, IDENT, RBRACE, SEMICOLON, EOF

Andentering that line at the REPL prints one token per line

Background

This is the first time the whole front of the pipeline runs together. The line let add = fn(x, y) { x + y }; exercises nearly everything you built: keywords (let, fn), identifiers, an assignment, delimiters, a binary operator, and the terminating semicolon. If the token stream comes out exactly as specified, your lexer is done.

Wiring it into the REPL turns that loop from an echo into a real tool: type a line, watch it decompose into tokens. Keep this token-printing mode around - it is the quickest way to see what the lexer thinks of any input, and it closes out the tokenizing chapter with something you can actually run.

Make it work

// in the REPL loop, replace the echo with a lexer pass
lex := NewLexer(line)
for tok := lex.NextToken(); tok.Type != EOF; tok = lex.NextToken() {
  fmt.Printf("{Type:%s Literal:%s}\n", tok.Type, tok.Literal)
}

CheckpointDONE

The REPL tokenizes whatever you type and prints the token stream - the lexer is complete. Commit and stop for today.