分词器会考虑在哪将文本切分成词（典型的如空格）的实际规则。

The tokenizers take care of the actual rules for where to break the text up into words (typically whitespace).

youdao

分词器会考虑在哪将文本切分成词（典型的如空格）的实际规则。

The tokenizers take care of the actual rules for where to break the text up into words (typically whitespace).

youdao