Some fields you may not want to be tokenized, for example a serial number field.

有些字段你可能不想它被分词，比如一连串的数字字段。

youdao

The contents of this file are tokenized into a list of strings and bound to a list called Rules.

文件的内容被分割放入一个字符串列表，然后绑定到名为Rules的变量。

youdao

The map workers (in the canonical example) split the work into individual vectors that contain the tokenized word and an initial value (1, in this case).

map 工作线程（在规范的示例中）将工作分割成包含已标记单词和初始值（在此情况下是 1）的单个矢量。

youdao

Once tokenized and hidden, any given card product is far more vulnerable to being displaced, as the issuers have already learned from PayPal.

FORBES: Tokenization And The Collapse Of The Credit Card Payment Model