Indexing - create an index of web crawl data.

索引——创建网页检索数据索引。

youdao

He gave an example of a web crawl they compressed with the system. The crawl contained 2.1B pages and the rows were named in the following form: “com.cnn.www/index.html:http”.

这个例子的蜘蛛 包含 2.1B 的页面，行按照以下的方式命名：“com.cnn.www/index.html:http”.在未压缩前的web page 页面大小是：45.1 TB ，压缩后的大小是：4.2 TB ， 只是原来的 9.2%。

youdao

Usually, users are forced to crawl the web or mine social-media sites to build their own.

通常，用户被迫抓取web或者挖掘社会媒体网站来建立他们自己的大型数据集。

youdao

We completed our Caffeine web indexing system last year, which allows us to crawl and index the web for fresh content quickly on an enormous scale.

FORBES: Google Search Now More Timely Than Ever

Called PageRank, it employs computer robots to crawl the Web for clues in content that can signal how websites are connected.

CNN: Why did Google's Eric Schmidt step down?

Lucky for you, we crawl the Web, dig into the news and bring you the Digital Download, a blog dedicated to the day's must-read tech stories.

FORBES: The Best Of The Digital Download