You can see that the home page has already been fetched and indexed, and the crawler is working on fetching and indexing the pages that have been linked directly from it.

从图中可以看到已经获取并索引了主页，程序现在的工作是获取和索引直接从该主页链接出的页面。

youdao

If you've removed a page or section, you can use the robot.txt to prevent the spiders crawling and indexing the broken links.

如果你删掉了一个页面或者网站的部分内容，你可以通过修改robot . txt预防蜘蛛爬行和索引那些无效链接。

youdao

Essentially, the process of indexing is identifying the words and expressions that best describe the page and assigning the page to particular keywords.

从本质上讲，在索引过程是确定词语最能描述网页的页面和分配特定的关键字。

youdao

Norvig's crew also aims to accelerate results by learning which irrelevant words (like "like") to discard when indexing a Web page.

FORBES

But detailed semantic indexing, which involves analyzing the meaning of every sentence in a page's text, is still a laborious process.

FORBES: Magazine Article