SQL text is just characters until you group them into tokens. Today you start the tokenizer by recognizing words - and telling a keyword like SELECT apart from a table name like users.
Split a run of letters into a token, and classify it as a keyword or an identifier, case-insensitively for keywords.
A parser never works on raw characters; it works on tokens - the words,
numbers, and symbols of the language. The tokenizer (or lexer) is the first
stage, and the first thing it must do is read a run of word characters and decide
what kind of word it is. A word starts with a letter and then runs on through
letters, digits, and underscores - so user_id and orders2 are each a single
identifier, which matters the moment real column names like user_id show up.
The distinction that matters is keyword versus identifier. SELECT and
FROM are reserved words with fixed meaning; users and name are names you
chose. SQL keywords are case-insensitive - select, SELECT, and Select are
the same word - so match them by upper-casing before checking your keyword set,
while preserving identifiers exactly as written. Start your keyword set small;
you will add to it as the parser grows.
var keywords = map[string]bool{"SELECT": true, "FROM": true /* ... */}// a word starts with a letter, then letters/digits/underscore:// read while isLetter(c) || isDigit(c) || c == '_'// if keywords[strings.ToUpper(word)] -> Keyword (store upper-cased)// else -> Identifier (store as written)