In this example, the newline character token is used to generate a kind of "multi-line" prompt.
在本例中，使用换行字符符号生成了一种“多行”提示符。
The second split method demonstrates using a different token character -- in this case, a colon -- to split a string into a sequence of strings.
第二个split方法演示如何使用不同的令牌字符（本例中使用的是冒号）将一个字符串分成一系列字符串。
The split method is useful because it splits a string into a sequence of smaller strings, using a token character (or any character in a given sequence of characters) as an indicator of where to chop.
split方法很有用，因为它可以将一个字符串分成几个较小的字符串序列，方法是将令牌字符(或给定字符序列中的任何字符)用作断开位置的指示器。
Any of the so-called control characters (those with a hex value below 0x20, the space character) can cause your parser to stop with an invalid token error.
所有所谓的控制字符(那些16进制值低于0x20的字符，空格字符)可能会导致您的解析器因为一个无效的标记错误而停止工作。
Next it loops for as long as it finds digits or letters and accumulates the character in a variable (token).
接下来，只要它发现有数字或字母，就一直循环下去，并将字符累积在变量（token ）中。
All others (i.e., those which can not be represented in the target character encoding) MUST be encoded using the ENTITY token.
其他的(如那些无法在目标字符编码表示的)则必须全部用ENTITY标记来编码。
Each line in the file will contain a word and its synonyms separated by a space character or some other known token.
文件中的每一行将包含一个单词及其同义词，并且使用空格或其他一些已知的标记分隔。
If such a character is found, it is the start of the first token.
如果这样一个人物被发现，它是第一个令牌开始。
By the same token, because he's handsome looks, good character, much welcome the fans.
同样，因为他的相貌英俊，人品很好，倍受球迷欢迎。
Defines a token for the signature that has the specified character array and signature length.
为具有指定字符数组和签名长度的签名定义标记。
White space can appear within a preprocessing token only as a part of a header name or between the quotation characters in a character literal or string literal.
仅当作为头名称的一部分，或在字符文字量或字符串文字量的引号字符之间时，空白才可以出现在预处理标记内部。
Effects of type frequency and token frequency on Chinese character recognition were explored in two experiments.
用识别整字和识别部件两种任务探讨两种声旁频率——组字频率和使用频率的作用。
Both the type frequency and the token frequency of phonetic component showed facilitating effect in character naming.
用识别整字和识别部件两种任务探讨两种声旁频率——组字频率和使用频率的作用。
The first character of a token must be the first alphanumeric character in the comment line.
语汇基元的第一个字元必须是在注解行的第一个英数字元。
Lexical analysis of the main tasks are: to scan source code character by character from left to right, resulting in one word (Token), also check the source of lexical errors.
词法分析的主要任务是：从左到右逐个字符地扫描源程序，产生一个个单词(Token)，同时检查源程序中的词法错误。
Lexical analysis of the main tasks are: to scan source code character by character from left to right, resulting in one word (Token), also check the source of lexical errors.
词法分析的主要任务是：从左到右逐个字符地扫描源程序，产生一个个单词(Token)，同时检查源程序中的词法错误。
应用推荐