These considerations relate to how your text is tokenized and then indexed by Full-Text Engine.
这些注意事项均与全文引擎如何对文本进行词汇切分再编制其索引有关。
The contents of this file are tokenized into a list of strings and bound to a list called Rules.
文件的内容被分割放入一个字符串列表，然后绑定到名为Rules的变量。
Erlang also gives us programmatic access to the tokenized, parsed and abstract forms of an expression.
Erlang还允许我们编程访问经过词法分析(tokenized)、语法分析之后、抽象形式的表达式。
A method for firming the BASIC user program with the tokenized source program is proposed in this paper.
本文提出了一种利用代号源程序固化BASIC用户程序的方法。
Should be specified as the string of hexadecimal digits representing the tokenized form of the public key.
应指定为十六进制数字的字符串以表示标记形式的公钥。
The SyncML encoding doesn't use them at all, but WML uses them for including tokenized body-text strings from a string table sent in the preamble.
这些在SyncML编码中根本没有使用，但是文档头部中发来一个字符串表，WML用上面那些元素处理经过标记的主体文本字符串。
The map workers (in the canonical example) split the work into individual vectors that contain the tokenized word and an initial value (1, in this case).
map工作线程（在规范的示例中）将工作分割成包含已标记单词和初始值（在此情况下是1）的单个矢量。
For example, if you tokenized a developerWorks article and sent it through an English-language parser, it might produce a tree structure like that shown in Figure 1.
例如，如果您将一篇 developerWorks 的文章做了标记并通过一个英语解析器将其发送，结果也许是如图1 中所示的树结构。
For <c:forTokens>, the string to be tokenized is specified through the items attribute, while the set of delimiters used to generate the tokens is provided through the delims attribute.
对于 <c:forTokens> 而言，通过items属性指定要标记化的字符串，而通过delims 属性提供用于生成标记的一组定界符。
For <c:forTokens>, the string to be tokenized is specified through the items attribute, while the set of delimiters used to generate the tokens is provided through the delims attribute.
对于 <c:forTokens> 而言，通过items属性指定要标记化的字符串，而通过delims 属性提供用于生成标记的一组定界符。
应用推荐