A contraction like "don't" is one word, not "don" plus "t". Today you teach the tokenizer to keep an apostrophe inside a word, but only when it sits between letters, so contractions survive and stray quotes do not.
Treat an apostrophe as part of a word when it falls between two letters, and as a separator otherwise.
English is full of contractions - don't, it's, we'll - and a tokenizer
that split them at the apostrophe would flag don and a lone t as misspellings
on every page. So the apostrophe gets one special rule: it stays inside a word,
but only when it is genuinely internal - a letter before it and a letter after
it.
That guard is what keeps the rule honest. A trailing apostrophe in James' has no
letter after it, so it ends the word and is dropped, giving James. A leading
quote has no letter before it, so it never starts a word. This is a small
refinement of the letter-run scan from before: letters always continue a word, an
apostrophe continues it conditionally, everything else breaks it.
// an apostrophe continues the current word ONLY if we are already// inside a word AND the next character is a letter; otherwise it// ends the word like any other separatorfunc isWordApostrophe(prevIsLetter bool, next rune) bool {// prevIsLetter && next is a letter}