At the same time, Web crawler developers work to create new crawling approaches that account for the ever-increasing complexity of Web pages, often with impact on their processing speed.

与此同时，web爬虫程序开发人员努力创建新的爬行方法以应对日益复杂的web页面，因为复杂的web页面常常会影响页面的处理速度。

youdao

Sitemaps 0.90 is an important step toward this goal and allows simple and efficient crawling even of complex and dynamic modern Web sites.

Sitemaps 0.90 是实现这一目标的重大突破，甚至在复杂和动态的现代 Web 站点中，它也能提供简单和有效的爬行。

youdao

This example demonstrates the crawling phase of a Web spider.

这个例子展示了Web spider爬行的阶段。

youdao

Radius has spent the past two and a half years crawling the Web to amass half a billion pieces of data on every small business in the United States.

FORBES: Radius: How Facebook's Youngest Hire Is Tackling The Local Data Problem

The problem I was trying to solve was a very specific problem of crawling the web, collecting all of these web pages and building indexes for them and maintaining them.

FORBES: Open-Source Solves Big-Data Problems: Talking to 'Mr. Hadoop,' Doug Cutting

The app essentially acts as a content hub for news, crawling the Web for top stories in key subjects and presenting them on the iPhone in summarized form, to fit a 4.5 inch screen.

FORBES: Teen Developer's Summly App Hits Half A Million Downloads; Publishers Come Knocking