The lexer is the part of our language knowledge that says "this is a sentence; this is punctuation; twenty-three is a single word."

词法分析器是我们的语言知识中识别“这是一个句子；这是标点；t wenty - three是一个单一的词”的那一部分。

youdao

Detect word boundaries more accurately by treating punctuation characters as parts of words.

检测单词边界时把标点视为单词的一部分。

youdao

Punctuation and common word frequency, which are not influenced by contents, can reflect the author's style, and the tests shows there are not significant differences in the two aspects.

在标点和普通词词频方面，不受内容影响，则可体现作者写作特点，数据检测表明在这两个方面不呈显著性差异。

youdao

Conventions in punctuation and word placement were all bent in this quest for pictorial mood, and this is what makes the arrival of both writers so startling in the history of prose.

FORBES: Magazine Article

In one particular extract, it says the paper omitted a punctuation mark, so that it appeared that the Koran was described with an obscene word, when this was not, in fact, the case.

BBC: Syrian book cleared of blasphemy