During the text segmentation, each word or token is determined for the lexical dictionary.
在文本分割期间，将针对词汇字典确定每个单词或标记。
If the tokenizer encounters neither token, it returns a lexical error.
如果记号赋予器未遇到这两个记号，则返回词法错误。
The generated lexical parser may read multiple characters ahead before it detects a token.
所生成的词法解析器可能会在检测到某个符号之前读入多个字符。
Since a lexical analyzer is actually nothing more than a parser optimized to process data into token streams, Spirit treats both parts of the process virtually identically.
由于词法分析器实际上只是优化，加工成令牌流数据分析器多，精神对待这一进程的两个部分几乎相同。
A preprocessing token is the minimal lexical element of the language in translation phases 3 through 6.
预处理标记是语言从第3到第6阶段中最小的词法元素。
Lexical analysis of the main tasks are: to scan source code character by character from left to right, resulting in one word (Token), also check the source of lexical errors.
词法分析的主要任务是：从左到右逐个字符地扫描源程序，产生一个个单词(Token)，同时检查源程序中的词法错误。
Lexical analysis of the main tasks are: to scan source code character by character from left to right, resulting in one word (Token), also check the source of lexical errors.
词法分析的主要任务是：从左到右逐个字符地扫描源程序，产生一个个单词(Token)，同时检查源程序中的词法错误。
应用推荐