If crawling the Web empirically and without commercial bias is the moral duty of a search engine, Baidu is, at least in the eyes of its competitors, not a moral company.

如果说不带任何商业偏见地抓取网站是一个搜索引擎的道德义务的话，那么，至少在竞争者眼中，百度不是一个有道德的公司。

youdao

Most search engines work by crawling the Web, indexing and filtering the content they find into massive databases, and searching these databases to find results matching a particular search query.

大多数搜索引擎的工作方式是：在Web上爬行，索引并过滤在海量数据库中找到的内容，然后搜索这些数据库以找到匹配特定搜索查询的结果。

youdao

But all the major and legitimate Web crawling engines obey the requests in robots.txt.

不过，所有主要的合法Web爬虫引擎都会遵从robots . txt内的要求。

youdao

Radius has spent the past two and a half years crawling the Web to amass half a billion pieces of data on every small business in the United States.

FORBES: Radius: How Facebook's Youngest Hire Is Tackling The Local Data Problem

The problem I was trying to solve was a very specific problem of crawling the web, collecting all of these web pages and building indexes for them and maintaining them.

FORBES: Open-Source Solves Big-Data Problems: Talking to 'Mr. Hadoop,' Doug Cutting

The app essentially acts as a content hub for news, crawling the Web for top stories in key subjects and presenting them on the iPhone in summarized form, to fit a 4.5 inch screen.

FORBES: Teen Developer's Summly App Hits Half A Million Downloads; Publishers Come Knocking