During the text segmentation, each word or token is determined for the lexical dictionary.
在文本分割期间，将针对词汇字典确定每个单词或标记。
If the tokenizer encounters neither token, it returns a lexical error.
如果记号赋予器未遇到这两个记号，则返回词法错误。
The generated lexical parser may read multiple characters ahead before it detects a token.
所生成的词法解析器可能会在检测到某个符号之前读入多个字符。
应用推荐