CMSC 214 - Project #1

C M S C 2 1 4
C o m p u t e r S c i e n c e I I
F a l l 2 0 0 2

Tokenizer Help

Everyone is making this tougher than it should be. Here's how things work for getLeftToken(). Assume you have a string str.

The character will fall in one of four categories:

It's a single quote
Begin searching for the next single quote. Extract the string from the first single quote to the second single quote. Do NOT do any processing of characters in between. In particular, do not modify blanks or double quotes, should they appear.
The token is all characters between the single quotes.
The rest of the string is all characters after the second single quote.
It's a double quote
Begin searching for the next double quote. Extract the string from the first double quote to the second double quote. Do NOT do any processing of characters in between. In particular, do not modify blanks or single quotes, should they appear.
The token is all characters between the double quotes.
The rest of the string is all characters after the second double quote.
It's neither
Begin searching for a blank space or until you reach the end of the string (use the size() of the string to help you out) whichever comes first.
Note: single quotes and double quotes will not appear when you do this in our tests.
The token is every non-blank character.
The rest of the string is all characters from the blank to the end.
You're already at the end of the string
This occurs when the string passed has 0 or more blanks.
Return the empty string as token, and the rest of the string is empty. split() should recognize that no token is empty, and therefore, not add it to the vector of strings.

See the class syllabus for policies concerning email
Last Modified: Fri Sep 13 20:11:45 EDT 2002