Some fields you may not want to be tokenized, for example a serial number field.
有些字段你可能不想它被分词，比如一连串的数字字段。
The contents of this file are tokenized into a list of strings and bound to a list called Rules.
文件的内容被分割放入一个字符串列表，然后绑定到名为Rules的变量。
The map workers (in the canonical example) split the work into individual vectors that contain the tokenized word and an initial value (1, in this case).
map工作线程（在规范的示例中）将工作分割成包含已标记单词和初始值（在此情况下是1）的单个矢量。
For <c:forTokens>, the string to be tokenized is specified through the items attribute, while the set of delimiters used to generate the tokens is provided through the delims attribute.
对于 <c:forTokens> 而言，通过items属性指定要标记化的字符串，而通过delims 属性提供用于生成标记的一组定界符。
These considerations relate to how your text is tokenized and then indexed by Full-Text Engine.
这些注意事项均与全文引擎如何对文本进行词汇切分再编制其索引有关。
Should be specified as the string of hexadecimal digits representing the tokenized form of the public key.
应指定为十六进制数字的字符串以表示标记形式的公钥。
Should be specified as the string of hexadecimal digits representing the tokenized form of the public key.
应指定为十六进制数字的字符串以表示标记形式的公钥。
应用推荐