Whitespace splits words, but quotes override that so a single argument can contain spaces. Today you rework the tokenizer to respect single and double quotes and to glue adjacent pieces into one word.
Split a line into words while treating quoted spans as literal, removing the quotes, and joining adjacent quoted and unquoted pieces.
Yesterday’s tokenizer split purely on whitespace, so there was no way to pass an
argument that contains a space. Quotes fix that: everything between a pair
of '...' or "..." is one literal chunk, spaces and all, and the quote
characters themselves are removed from the result. So echo 'hello world' passes
echo a single argument, hello world.
The subtle rule is that quoting is about spans, not whole words. A word can be
built from several adjacent pieces - quoted and unquoted - with no space between
them, and they all fuse into one: a"b"c is the single word abc, and
"$HOME"/bin will later be one word too. So think in terms of “am I currently
inside a quote?” as you scan character by character, ending a word only on
unquoted whitespace. Single and double quotes behave the same for grouping
today; their difference - whether $ expands inside them - arrives once expansion
does.
// build one word char by char; a quote flips a "quoted" mode// inside quotes, spaces are literal; the quote chars themselves are droppedif (c == '\'' || c == '"') { in_quote = c; continue; } // openingif (c == in_quote) { in_quote = 0; continue; } // closingif (!in_quote && isspace(c)) end_word();else append_char(c);