自动检测——对机器人、爬虫、扫描及其他恶意行为进行检测。

Automation detection - Detecting bots, crawlers, scanners and other surface malicious activity.

youdao

在后端流程中，网络爬虫或者机器人从因特网上获取web页面，然后索引子系统解析这些Web页面并存入索引文件中。

In the back-end process, a spider or robot fetches the Web pages from the Internet, and then the indexing subsystem parses the Web pages and stores them into the index files.

youdao

该文章建议阻止Googlebot （Google的搜索机器人软件）爬虫收录网站RSS/Atom feeds的站长们通过robots.txt文件解除爬虫限制。

The post advises website owners who are blocking Google's search bot software known as Googlebot from crawling their RSS/Atom feeds to unblock it via their robots.txt file.

youdao