You can use a shorter or longer timeout based on how long you want the program to run, your network connection's speed, and the depth of the Web site you want to crawl.

根据想要让程序运行多久、网络连接速度和想对网站进行处理的深度，您可以使用更长或更短的超时。

youdao

To take this program to the next level and make it crawl a whole site, it's best to break it apart into smaller tasks.

要改进此程序，使其作用到整个网站，最好将其分解为小一些的任务。

youdao

In this command, the -r switch tells GNU Wget to recursively follow all links found on that page, so the tool will crawl the entire site.

在这个命令中，- r开关告诉GNU Wget递归地跟踪那个页面上的所有链接，所以该工具将抓取整个站点。

youdao

The flood of messages slowed Twitter to a crawl at times or produced error messages saying the site was over capacity.

CNN: Mark Milian,