You can see that the home page has already been fetched and indexed, and the crawler is working on fetching and indexing the pages that have been linked directly from it.

从图中可以看到已经获取并索引了主页，程序现在的工作是获取和索引直接从该主页链接出的页面。

youdao

If you've removed a page or section, you can use the robot.txt to prevent the spiders crawling and indexing the broken links.

如果你删掉了一个页面或者网站的部分内容，你可以通过修改robot . txt预防蜘蛛爬行和索引那些无效链接。

youdao

Norvig's crew also aims to accelerate results by learning which irrelevant words (like "like") to discard when indexing a Web page.

FORBES

But detailed semantic indexing, which involves analyzing the meaning of every sentence in a page's text, is still a laborious process.

FORBES: Magazine Article