Why do I need a tokenizer for each language?
为什么我需要为每种语言的标记？
With the tokenizer, it is easy to decode the processing instructions.
有了 tokenizer，就很容易译码处理指令。
If the tokenizer encounters neither token, it returns a lexical error.
如果记号赋予器未遇到这两个记号，则返回词法错误。
A good lexer example can help a lot with learning how to write a tokenizer.
一个好的lexer例子会非常有助于学习如何编写断词器（tokenizer）。
What the parser fills in depends on the path it takes through the tokenizer.
解析器填入的内容取决于它通过记号赋予器所采用的路径。
Fortunately, gnosis.indexer.TextSplitter is a robust tokenizer. For example.
幸运的是，gnosis . indexer . TextSplitter是一个健壮的断词工具。
Not surprisingly, the part of the parser that's responsible for that task is called the scanner or tokenizer.
解析器中负责该任务的部件称为扫描器(scanner)或记号赋予器(tokenizer)一点也不稀奇。
The process that scans an input string and determines what the tokens are is called the tokenizer. In the query.
扫描输入字符串以及判断是何标记的过程称作记号赋予器(tokenizer)。
The tokenizer moves through the stream, inspecting characters one at a time until it encounters either an integer or the end of file.
记号赋予器穿过输入流，每次检查一个字符，直到它遇到一个整数或者直至文件结束。
Quantifiers can be used within the regular expressions of the Spark tokenizer, and can be simulated by recursion in parse expression grammars.
计量符可以在Spark记号赋予器(tokenizer)的正则表达式中使用，并可以用解析表达式语法中的递归来进行模拟。
The analysis screen displays the result of each term after it has been processed by the Tokenizer or TokenFilter named above the table results.
分析屏幕显示了每个条件在被上述表结果Tokenizer或TokenFilter处理后的结果。
It is important to preserve this ordering because the order that tokenizer rules appear in the file are the order in which token rules are applied.
保持该次序是极其重要的，因为记号赋予器规则在文件中出现的次序就是应用标记规则的次序。
Readers with some experience working with parsers will probably notice how similar this is to tokenizer generated by a "lex" style programming tool.
具有解析器实践经验的读者可能会注意到，这和由“lex”风格的编程工具生成的标记器是很相似的。
However, I believe that if one needed to build a tokenizer and wanted to avoid the complications of using another programming tool one could build an active pattern that fulfilled this requirement.
然而，我相信如果一个人需要构建一个标记器并想避免由于使用其他编程工具所带来的复杂性的话，那么他可以构建一个活动模式来满足这样的需求。
However, I believe that if one needed to build a tokenizer and wanted to avoid the complications of using another programming tool one could build an active pattern that fulfilled this requirement.
然而，我相信如果一个人需要构建一个标记器并想避免由于使用其他编程工具所带来的复杂性的话，那么他可以构建一个活动模式来满足这样的需求。
应用推荐