It is important to note that OmniFind is installed in a multiple-server configuration, in which the search crawler component does not reside on a machine that has an application server.

特别注意，OmniFind被安装在一个多服务器配置中，在该配置中，搜索crawler组件没有位于拥有应用服务器的机器上。

youdao

Today, the approach that many search engines still recommend for some level of crawling efficiency is to make an HTML-based site map page available for a crawler to pick up.

如今，许多搜索引擎仍然推荐实现某一级别爬行效率的方法，那就是使爬虫程序可以爬行基于 HTML 的站点地图页面。

youdao

Each search engine has its own automated program called a "web spider" or "web crawler" that crawls the web.

每个搜索引擎都有自己爬行网页的自动化程序，叫做“网络蜘蛛（web spider）”或“网络爬虫（web crawler）”。

youdao

Even though the sensitive information was hidden in the HTML and not visible in plain view, the Google crawler observed it and recorded the information to put into its search index.

FORBES: Blippy Apologizes For Google-Searchable Credit Cards, Announces New Security Plan