The filters do any post-tokenizing work on the tokens (typically dropping out punctuation and commonly occurring words like "the", "an", "a", etc).

滤词器在分词之后对标号做进一步处理（最典型的比如去掉标点符号和一些常见词，像"the", "an", "a"）。

youdao

The filters do any post-tokenizing work on the tokens (typically dropping out punctuation and commonly occurring words like "the", "an", "a", etc).

滤词器在分词之后对标号做进一步处理（最典型的比如去掉标点符号和一些常见词，像"the", "an", "a"）。

youdao