If the tokenizer encounters neither token, it returns a lexical error.
如果记号赋予器未遇到这两个记号，则返回词法错误。
A good lexer example can help a lot with learning how to write a tokenizer.
一个好的lexer例子会非常有助于学习如何编写断词器（tokenizer）。
What the parser fills in depends on the path it takes through the tokenizer.
解析器填入的内容取决于它通过记号赋予器所采用的路径。
Not surprisingly, the part of the parser that's responsible for that task is called the scanner or tokenizer.
解析器中负责该任务的部件称为扫描器(scanner)或记号赋予器(tokenizer)一点也不稀奇。
The tokenizer moves through the stream, inspecting characters one at a time until it encounters either an integer or the end of file.
记号赋予器穿过输入流，每次检查一个字符，直到它遇到一个整数或者直至文件结束。
Quantifiers can be used within the regular expressions of the Spark tokenizer, and can be simulated by recursion in parse expression grammars.
计量符可以在Spark记号赋予器(tokenizer)的正则表达式中使用，并可以用解析表达式语法中的递归来进行模拟。
It is important to preserve this ordering because the order that tokenizer rules appear in the file are the order in which token rules are applied.
保持该次序是极其重要的，因为记号赋予器规则在文件中出现的次序就是应用标记规则的次序。
Readers with some experience working with parsers will probably notice how similar this is to tokenizer generated by a "lex" style programming tool.
具有解析器实践经验的读者可能会注意到，这和由“lex”风格的编程工具生成的标记器是很相似的。
Readers with some experience working with parsers will probably notice how similar this is to tokenizer generated by a "lex" style programming tool.
具有解析器实践经验的读者可能会注意到，这和由“lex”风格的编程工具生成的标记器是很相似的。
应用推荐