Most search engines work by crawling the Web, indexing and filtering the content they find into massive databases, and searching these databases to find results matching a particular search query.

大多数搜索引擎的工作方式是：在Web上爬行，索引并过滤在海量数据库中找到的内容，然后搜索这些数据库以找到匹配特定搜索查询的结果。

youdao

But all the major and legitimate Web crawling engines obey the requests in robots.txt.

不过，所有主要的合法Web爬虫引擎都会遵从robots . txt内的要求。

youdao

This example demonstrates the crawling phase of a Web spider.

这个例子展示了Web spider爬行的阶段。

youdao

Radius has spent the past two and a half years crawling the Web to amass half a billion pieces of data on every small business in the United States.

FORBES: Radius: How Facebook's Youngest Hire Is Tackling The Local Data Problem

The problem I was trying to solve was a very specific problem of crawling the web, collecting all of these web pages and building indexes for them and maintaining them.

FORBES: Open-Source Solves Big-Data Problems: Talking to 'Mr. Hadoop,' Doug Cutting

The app essentially acts as a content hub for news, crawling the Web for top stories in key subjects and presenting them on the iPhone in summarized form, to fit a 4.5 inch screen.

FORBES: Teen Developer's Summly App Hits Half A Million Downloads; Publishers Come Knocking