词法约定(表示IDL文件的关键字、注释和文字的字符记号)规定ISO 8859.1字符集表示IDL文件中的字符。
The lexical conventions (character tokens that will represent the keywords, comments, and literals of an IDL file) stipulate that the ISO 8859.1 character set represent the characters in an IDL file.
记号赋予器穿过输入流，每次检查一个字符，直到它遇到一个整数或者直至文件结束。
The tokenizer moves through the stream, inspecting characters one at a time until it encounters either an integer or the end of file.
扫描程序读取缓冲中的单个字符并返回解析器能理解的记号(称做标记)——如关键字、操作符及针对特定语言的标点符号。
A scanner reads in the individual characters of a buffer and returns symbols (called tokens) that the parser will understand — such as keywords, operators, and language-specific punctuation.
这些数字将会在可能的合法字符范围之外；这样，lexer可以原样返回单个字符本身，或者返回使用这些定义值的记号。
These Numbers will be outside the range of possible valid characters; thus, the lexer can return individual characters as themselves, or special tokens using these defined values.
我可以在查询字符串中放入一个记号（[value*]），然后在查询结果可以被访问的时候，创建一个用来得到cutoffDate值的函数。
What I’m doing is placing a token ([value*]) in the query string and then creating a function that I can use to get the value of cutoffDate when the results of the query are accessed.
扫描输入字符串以及判断是何标记的过程称作记号赋予器(tokenizer)。
The process that scans an input string and determines what the tokens are is called the tokenizer. In the query.
画押，记号用来代替签名的标记，如十字符。
画押，记号：用来代替签名的标记，如十字符。
你可以随意扩展这个解决方案，让记号可以用某组字符以外的任何字符分割，这组字符可以作为一个串参数传递。
Optionally, you may extend the solution so that tokens may be separated by any character out of a set of character given as a string argument.
记号指字符串的一个子串，它位于字符串的空格之间，或者位于空格和串尾之间。
A token is a substring occurring between space characters in the string or between space characters and the end of the string.
与取得列表中的元素一样，也可以通过下标记号取得字符串中的某个字符。
Just like getting individual items out of a list, you can get individual characters out of a string using index notation.
与取得列表中的元素一样，也可以通过下标记号取得字符串中的某个字符。
Just like getting individual items out of a list, you can get individual characters out of a string using index notation.
应用推荐