Each line in the file will contain a word and its synonyms separated by a space character or some other known token.
文件中的每一行将包含一个单词及其同义词，并且使用空格或其他一些已知的标记分隔。
During the text segmentation, each word or token is determined for the lexical dictionary.
在文本分割期间，将针对词汇字典确定每个单词或标记。
The synonyms and the word would be separated by a known token.
该单词及其同义词使用一个已知的标记分隔。
Each token (word) is then emitted to stdout with an associated value of 1 (separated by a TAB).
使用为1的相关值(通过选项卡分隔)将每一个标记(单词)发送到stdout。
A lexeme is a string, just like a token, but it has been normalized so that different forms of the same word are made alike.
语意是一个字串，和记号很像，但是它已经是一般化了的，这样同一个词的不同形式就可以一样来处理。
Lexical analysis of the main tasks are: to scan source code character by character from left to right, resulting in one word (Token), also check the source of lexical errors.
词法分析的主要任务是：从左到右逐个字符地扫描源程序，产生一个个单词(Token)，同时检查源程序中的词法错误。
So many larvae die of being worn out when they strive out, the word 'flying' being their everlasting tragical token.
太多的幼虫在往外冲杀的时候力竭身亡，不幸成了“飞翔”这个词的悲壮祭品。
He touched her tray with the tip of his finger as a token of acceptance, and went down the street without a word.
他用指尖轻碰了一下她的盘子以示受礼，然后一言不发地沿街道走去。
And they said no word at all, but only cast themselves down before him, and touched the hem of his garment in token of their healing.
他们默默不语，只是俯在他的面前，触碰他的衣边，示意他们已经痊愈。
He touched her tray with the tip of his finger as a token of acceptance, and went down the street without a word.
他用指尖碰了下她的托盘以示接受，然后一言不发，沿着街道走去。
He touched her tray with the tip of his finger as a token of acceptance, and went down the street without a word.
他用指尖碰了下她的托盘以示接受，然后一言不发，沿着街道走去。
应用推荐