扫描输入字符串以及判断是何标记的过程称作记号赋予器(tokenizer)。

The process that scans an input string and determines what the tokens are is called the tokenizer. In the query.

youdao

扫描输入字符串以及判断是何标记的过程称作记号赋予器(tokenizer)。

The process that scans an input string and determines what the tokens are is called the tokenizer. In the query.

youdao