To tokenize (that is, split a line into word-length pieces) a string, use the split() function.

若要 tokenize（也就是说，将一行拆分为单词长度的片段）某个字符串，请使用split() 函数。

youdao

Tokenize Configuration, which provides a repeatable way of injecting variable information into configuration files

Tokenize Configuration，这个模式提供一种可重复的把可变信息注入配置文件中的方法

youdao

The new tokenize function performs the opposite action, disaggregating a string with a separator to mark the boundaries.

新增的 tokenize函数执行相反的操作，用分隔符标记字符串之间的边界。

youdao

You can control white space recognition (that separates parse symbols) by defining the reserved d_whitespace() function; this enables you to tokenize however you like.

您可以通过定义保留的 d_whitespace()函数来控制空格的识别（它分离解析符号）；这样就使得您可以随意使用标记。

youdao

You can control white space recognition (that separates parse symbols) by defining the reserved d_whitespace() function; this enables you to tokenize however you like.

您可以通过定义保留的 d_whitespace()函数来控制空格的识别（它分离解析符号）；这样就使得您可以随意使用标记。

youdao