Indexing - create an index of web crawl data.

索引——创建网页检索数据索引。

youdao

Sitemaps 0.90 is a simple and intuitive way for webmasters to provide the right level of information to Web crawlers so that they can efficiently crawl a Web site.

Sitemaps 0.90为 Web 站点管理员提供了简单而且直观的方法，使他们可以向 web 爬虫程序提供正确的信息，从而使爬虫程序能够高效地爬行某个 Web 站点。

youdao

He gave an example of a web crawl they compressed with the system. The crawl contained 2.1B pages and the rows were named in the following form: “com.cnn.www/index.html:http”.

这个例子的蜘蛛 包含 2.1B 的页面，行按照以下的方式命名：“com.cnn.www/index.html:http”.在未压缩前的web page 页面大小是：45.1 TB ，压缩后的大小是：4.2 TB ， 只是原来的 9.2%。

youdao

We completed our Caffeine web indexing system last year, which allows us to crawl and index the web for fresh content quickly on an enormous scale.

FORBES: Google Search Now More Timely Than Ever

Called PageRank, it employs computer robots to crawl the Web for clues in content that can signal how websites are connected.

CNN: Why did Google's Eric Schmidt step down?

Lucky for you, we crawl the Web, dig into the news and bring you the Digital Download, a blog dedicated to the day's must-read tech stories.

FORBES: The Best Of The Digital Download