tokenize

Generates a List of Token from a String.

The accepted Token are as follows:

• Single quote delimited strings with escapable single quotes terminating on end of line and end of string,

• Double quote delimited strings with escapable double quotes terminating on end of line and end of string,

• Semicolon signaled line comments,

• Brackets: (, ), {, }, [, ], and

• Whitespace separated unicode-16 strings.

Whitespace is considered insignificant and is only used to mark separation of tokens.

This method may report the following TokenizerError:

• MalformedStringError: the closing quote does not appear on the same line as the opening quote or does not exist at all.

Return

Either the first TokenizerError reported during tokenization or the List of Token produced from input.

Parameters