This process is known more formally as parsing (or to be more precise, tokenizing, lexing, and parsing).

这个过程更正式的称呼是解析(parsing)(更准确地说，是标记解释(tokenizing)、词法解析(lexing)和语法解析)。

youdao

At the level of tokenizing, we are not yet interested in how the tokens are structured, just what they are.

在记号赋予这一层，我们还不想讨论如何构造记号，让它们维持原样就可以了。

youdao

Tokenizing/Lexing: breaking up a string of characters into meaningful (to the language) chunks, called tokens.

分词/词法分析：分词指将一段字符串破成有意义的（相对于这门语言）小块。

youdao