When spidering or scraping, always obey the robots.txt file available on the server and incorporate it into your politeness policy.

在进行这种操作时，一定要遵循服务器上提供的robots. txt文件的指示，并将其结合到您的礼貌原则当中。

youdao

The behavior policies define which pages the crawler will bring down to the indexer, how often to go back to a Web site to check it again, and something called a politeness policy.

这种行为策略定义了爬虫会将哪些页面带入索引程序、以什么样的频率回到Web站点上再次对它进行检查，以及一种礼貌原则。

youdao

The behavior policies define which pages the crawler will bring down to the indexer, how often to go back to a Web site to check it again, and something called a politeness policy.

这种行为策略定义了爬虫会将哪些页面带入索引程序、以什么样的频率回到Web站点上再次对它进行检查，以及一种礼貌原则。

youdao